Updated logging

README.md

@@ -1,10 +1,19 @@
 # Nomad Network - Communicate Freely
 
+> [!WARNING]  
+> Several hastily launched, incorrect, and in most cases entirely LLM-generated fakes of Nomad Network and associated tools are currently being circulated and marketed. Most of these "projects" violate the open source GPLv3 license that Nomad Network was published under, but claim independent ownership and license grants. Such claims or grants are **not** legally valid, and **not** recognized by the Nomad Network authors and copyright holders.
+>
+> Any claimed assertion of copyright or grant of license that such projects are making are [void grants](https://reticulum.network/manual/brandolinis.html#void-grants-legal-foundations-of-machine-generated-code). Both the violators themselves, and **all** downstream projects using the infringing derivatives, are **directly** and **personally** liable for the legal consequences.
+>
+> For more detailed information, see the relevant sections of the [Brandolini's Reference](https://reticulum.network/manual/brandolinis.html) chapter of the Reticulum Manual. For software and projects recognized by the Reticulum community, see the [Programs Using Reticulum](https://reticulum.network/manual/software.html) chapter.
+>
+> Do **not** use these "projects", they are a hazard to the entire ecosystem we have been carefully building over the last ten years.
+
 *This repository is [a public mirror](./MIRROR.md). All development is happening elsewhere.*
 
 Off-grid, resilient mesh communication with strong encryption, forward secrecy and extreme privacy.
 
-![Screenshot](https://github.com/markqvist/NomadNet/raw/master/docs/screenshots/1.png)
+![Screenshot](https://github.com/markqvist/NomadNet/raw/master/docs/screenshots/1a.png)
 
 Nomad Network allows you to build private and resilient communications platforms that are in complete control and ownership of the people that use them. No signups, no agreements, no handover of any data, no permissions and gatekeepers.
 
@@ -12,15 +21,23 @@ Nomad Network is build on [LXMF](https://github.com/markqvist/LXMF) and [Reticul
 
 Nomad Network does not need any connections to the public internet to work. In fact, it doesn't even need an IP or Ethernet network. You can use it entirely over packet radio, LoRa or even serial lines. But if you wish, you can bridge islanded networks over the Internet or private ethernet networks, or you can build networks running completely over the Internet. The choice is yours. Since Nomad Network uses Reticulum, it is efficient enough to run even over *extremely* low-bandwidth medium, and has been succesfully used over 300bps radio links.
 
-If you'd rather want to use an LXMF client with a graphical user interface, you may want to take a look at [Sideband](https://github.com/markqvist/sideband), which is available for Linux, Android and macOS.
+If you'd rather want to use an LXMF client with a graphical user interface, you may want to take a look at [Sideband](https://github.com/markqvist/sideband), which is available for Linux, Android, Windows and macOS.
+
+Other Nomad Network clients include:
+
+- [MeshChatX](https://meshchatx.com/)
+- [Columba (mobile)](https://github.com/torlando-tech/columba)
+- [Ren Browser](https://github.com/Quad4-Software/Ren-Browser)
 
 ## Notable Features
  - Encrypted messaging over packet-radio, LoRa, WiFi or anything else [Reticulum](https://github.com/markqvist/Reticulum) supports.
  - Zero-configuration, minimal-infrastructure mesh communication
  - Distributed and encrypted message store holds messages for offline users
  - Connectable nodes that can host pages and files
+ - Image rendering support in terminals that support it
  - Node-side generated pages with PHP, Python, bash or others
  - Built-in text-based browser for interacting with contents on nodes
+ - Built-in RRC client for live, many-to-many chat
  - An easy to use and bandwidth efficient markup language for writing pages
  - Page caching in browser
 
@@ -76,10 +93,10 @@ The first time the program is running, you will be presented with the **Guide se
 
 To use Nomad Network on packet radio or LoRa, you will need to configure your Reticulum installation to use any relevant packet radio TNCs or LoRa devices on your system. See the [Reticulum documentation](https://markqvist.github.io/Reticulum/manual/interfaces.html) for info. For a general introduction on how to set up such a system, take a look at [this post](https://unsigned.io/private-messaging-over-lora/).
 
-If you want to try Nomad Network without building your own physical network, you can connect to the [Unsigned.io RNS Testnet](https://github.com/markqvist/Reticulum#public-testnet) over the Internet, where there is already some Nomad Network and LXMF activity. If you connect to the testnet, you can leave nomadnet running for a while and wait for it to receive announces from other nodes on the network that host pages or services, or you can try connecting directly to some nodes listed here:
+If you want to try Nomad Network without building your own physical network, you can connect to the [distributed backbone](https://reticulum.network/manual/gettingstartedfast.html#bootstrapping-connectivity) over the Internet, where there is already quite a bit of Nomad Network and LXMF activity. If you connect to the testnet, you can leave nomadnet running for a while and wait for it to receive announces from other nodes on the network that host pages or services, or you can try connecting directly to some nodes listed here:
 
- - `abb3ebcd03cb2388a838e70c001291f9` Dublin Hub Testnet Node
+ - `9ce92808be498e9e05590ff27cbfdfe4` The rns.recipes forum
- - `ea6a715f814bdc37e56f80c34da6ad51` Frankfurt Hub Testnet Node
+ - `a4a5e861626ce97c9aa544d9ecdf6d22` rmap.world
 
 To browse pages on a node that is not currently known, open the URL dialog in the `Network` section of the program by pressing `Ctrl+U`, paste or enter the address and select `Go` or press enter. Nomadnet will attempt to discover and connect to the requested node.
 
@@ -88,42 +105,12 @@ You can install Nomad Network on Android using Termux, but there's a few more co
 
 For a native Android application with a graphical user interface, have a look at [Sideband](https://github.com/markqvist/Sideband).
 
-### Docker Images
-
-Nomad Network is automatically published as a docker image on Github Packages. Image tags are one of either `master` (for the very latest commit) or the version number (eg `0.2.0`) for a specific release.
-
-```sh
-$ docker pull ghcr.io/markqvist/nomadnet:master
-
-# Run nomadnet interactively in a container
-$ docker run -it ghcr.io/markqvist/nomadnet:master --textui
-
-# Run nomadnet as a daemon, using config stored on the host machine in specified
-# directories, and connect the containers network to the host network (which will
-# allow the default AutoInterface to automatically peer with other discovered
-# Reticulum instances).
-$ docker run -d \
-  -v /local/path/nomadnetconfigdir/:/root/.nomadnetwork/ \
-  -v /local/path/reticulumconfigdir/:/root/.reticulum/ \
-  --network host
-  ghcr.io/markqvist/nomadnet:master
-
-# You can also keep the network of the container isolated from the host, but you
-# will need to manually configure one or more Reticulum interfaces to reach other
-# nodes in a network, by editing the Reticulum configuration file.
-$ docker run -d \
-  -v /local/path/nomadnetconfigdir/:/root/.nomadnetwork/ \
-  -v /local/path/reticulumconfigdir/:/root/.reticulum/ \
-  ghcr.io/markqvist/nomadnet:master
-
-# Send daemon log output to console instead of file
-$ docker run -i ghcr.io/markqvist/nomadnet:master --daemon --console
-```
 
 ## Tools & Extensions
 
 Nomad Network is a very flexible and extensible platform, and a variety of community-provided tools, utilities and node-side extensions exist:
 
+- [Retipedia](https://github.com/RFnexus/Retipedia)
 - [NomadForum](https://codeberg.org/AutumnSpark1226/nomadForum) ([GitHub mirror](https://github.com/AutumnSpark1226/nomadForum))
 - [NomadForecast](https://github.com/faragher/NomadForecast)
 - [micron-blog](https://github.com/randogoth/micron-blog)
@@ -137,9 +124,6 @@ Nomad Network is a very flexible and extensible platform, and a variety of commu
 - [POPR](https://github.com/faragher/POPR)
 - [LXMF Tools](https://github.com/SebastianObi/LXMF-Tools)
 
-## Help & Discussion
-
-For help requests, discussion, sharing ideas or anything else related to Nomad Network, please have a look at the [Nomad Network discussions pages](https://github.com/markqvist/Reticulum/discussions/categories/nomad-network).
 
 ## Support Nomad Network
 You can help support the continued development of open, free and private communications systems by donating via one of the following channels:
@@ -161,32 +145,12 @@ You can help support the continued development of open, free and private communi
 - Ko-Fi: https://ko-fi.com/markqvist
 
 
-## Development Roadmap
-
-- New major features
-    - Network-wide propagated bulletins and discussion threads
-    - Collaborative maps and geospatial information sharing
-- Minor improvements and fixes
-    - Link status (RSSI and SNR) in conversation or conv list
-    - Ctrl-M shorcut for jumping to menu
-    - Share node with other users / send node info to user
-    - Fix internal editor failing on some OSes with no "editor" alias
-    - Possibly add a required-width header
-    - Improve browser handling of remote link close
-    - Better navigation handling when requests fail (also because of closed links)
-    - Retry failed messages mechanism
-    - Re-arrange buttons to be more consistent
-    - Term compatibility notice in readme
-    - Selected icon in conversation list
-    - Possibly a Search Local Nodes function
-    - Possibly add via entry in node info box, next to distance
-
 ## Caveat Emptor
-Nomad Network is beta software, and should be considered as such. While it has been built with cryptography best-practices very foremost in mind, it _has not_ been externally security audited, and there could very well be privacy-breaking bugs. If you want to help out, or help sponsor an audit, please do get in touch.
+Nomad Network is beta software, and should be considered as such. Use at your own risk and responsibility.
 
 ## Screenshots
 
-![Screenshot 1](https://github.com/markqvist/NomadNet/raw/master/docs/screenshots/1.png)
+![Screenshot 1](https://github.com/markqvist/NomadNet/raw/master/docs/screenshots/1a.png)
 
 ![Screenshot 2](https://github.com/markqvist/NomadNet/raw/master/docs/screenshots/2.png)
 
@@ -194,4 +158,6 @@ Nomad Network is beta software, and should be considered as such. While it has b
 
 ![Screenshot 4](https://github.com/markqvist/NomadNet/raw/master/docs/screenshots/4.png)
 
-![Screenshot 5](https://github.com/markqvist/NomadNet/raw/master/docs/screenshots/5.png)
+![Screenshot 5](https://github.com/markqvist/NomadNet/raw/master/docs/screenshots/5.png)
+
+![Screenshot 6](https://github.com/markqvist/NomadNet/raw/master/docs/screenshots/6.png)

README.mu

@@ -16,6 +16,7 @@ If you'd rather want to use an LXMF client with a graphical user interface, you
  • Zero-configuration, minimal-infrastructure mesh communication
  • Distributed and encrypted message store holds messages for offline users
  • Connectable nodes that can host pages and files
+ • Image rendering support in terminals that support it
  • Node-side generated pages with PHP, Python, bash or others
  • Built-in text-based browser for interacting with contents on nodes
  • Built-in RRC client for live, many-to-many chat
@@ -119,4 +120,4 @@ For this to be possible, I need your help. Please support the continued developm
 
 >> Caveat Emptor
 
-Nomad Network is experimental software, and should be considered as such. While it has been built with cryptography best-practices very foremost in mind, it `!has not`! been externally security audited, and there could very well be privacy-breaking bugs. Use at your own risk and responsibility.
+Nomad Network is experimental software, and should be considered as such. Use at your own risk and responsibility.

nomadnet/Directory.py

@@ -6,7 +6,7 @@ import nomadnet
 import threading
 import RNS.vendor.umsgpack as msgpack
 
-from LXMF import pn_announce_data_is_valid
+from LXMF import pn_announce_data_is_valid, PN_META_IMPL_NAME, PN_META_VERSION
 from nomadnet.util import strip_modifiers
 from nomadnet.util import sanitize_name
 
@@ -21,7 +21,10 @@ class PNAnnounceHandler:
                 data = msgpack.unpackb(app_data)
 
                 if data[2] == True:
-                    RNS.log("Received active propagation node announce from "+RNS.prettyhexrep(destination_hash), RNS.LOG_DEBUG)
+                    metadata = data[6]
+                    if not PN_META_IMPL_NAME in metadata or not PN_META_VERSION in metadata: impl_str = ""
+                    else: impl_str = f" running {metadata[PN_META_IMPL_NAME]} {metadata[PN_META_VERSION]}"
+                    RNS.log(f"Received active propagation node announce from {RNS.prettyhexrep(destination_hash)}{impl_str}", RNS.LOG_DEBUG)
 
                     associated_peer = RNS.Destination.hash_from_name_and_identity("lxmf.delivery", announced_identity)
                     associated_node = RNS.Destination.hash_from_name_and_identity("nomadnetwork.node", announced_identity)

nomadnet/Node.py

@@ -4,9 +4,13 @@ import sys
 import RNS
 import time
 import threading
+import hashlib
+import shutil
 import subprocess
 import RNS.vendor.umsgpack as msgpack
 
+from RNS.Utilities.rngit.media import convert_file_to_webp
+
 class Node:
     JOB_INTERVAL = 5
     SETTINGS_FLUSH_INTERVAL = 60
@@ -26,6 +30,7 @@ class Node:
         self.job_interval = Node.JOB_INTERVAL
         self.should_run_jobs = True
         self.settings_dirty = False
+        self.conversion_cache_lock = threading.Lock()
         self.last_settings_flush = time.time()
         self.app_data = None
         self.name = self.app.node_name
@@ -158,7 +163,12 @@ class Node:
             RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
             return None
 
-    MEDIA_EXTS = [".webp"]
+    NATIVE_MEDIA_EXTS = [".webp"]
+    MEDIA_EXTS = [".webp", ".png", ".jpg", ".jpeg", ".bmp", ".gif", ".tiff"]
+    CONVERSION_QUALITY = 85
+    CONVERSION_MAX_DIMENSION = 1200
+    CONVERSION_CACHE_MAX_FILES = 256
+    CONVERSION_CACHE_MAX_BYTES = 192*1024*1024
     def serve_media(self, path, data, request_id, link_id, remote_identity, requested_at):
         if not type(data) == dict: return None
         if not "path" in data: return None
@@ -191,8 +201,17 @@ class Node:
         request_allowed = self.request_allowed(media_path, remote_identity)
         try:
             if request_allowed:
-                file_path = media_path
+                if not base_ext.lower() in self.NATIVE_MEDIA_EXTS:
-                file_name = os.path.basename(file_path)
+                    converted_path = self.convert_media_to_webp(media_path)
+                    if converted_path is False:
+                        RNS.log(f"Could not convert {media_path} to WebP", RNS.LOG_DEBUG)
+                        return False
+                    file_path = converted_path
+                    file_name = os.path.splitext(base_name)[0]+".webp"
+                else:
+                    file_path = media_path
+                    file_name = base_name
+
                 RNS.log(f"Serving media: {file_path}", RNS.LOG_VERBOSE)
                 return [open(file_path, "rb"), {"name": file_name.encode("utf-8")}]
             
@@ -205,6 +224,65 @@ class Node:
             RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
             return False
 
+    def convert_media_to_webp(self, media_path, quality = CONVERSION_QUALITY, max_dimension = CONVERSION_MAX_DIMENSION):
+        try:
+            hasher = hashlib.sha256()
+            with open(media_path, "rb") as fh:
+                for chunk in iter(lambda: fh.read(65536), b""): hasher.update(chunk)
+            source_hash = hasher.hexdigest()
+        except Exception as e:
+            RNS.log(f"Could not read media source for conversion: {media_path}", RNS.LOG_WARNING)
+            return False
+
+        if quality == None: quality = Node.CONVERSION_QUALITY
+        if max_dimension == None: max_dimension = Node.CONVERSION_MAX_DIMENSION
+
+        cache_name  = f"{source_hash}.q{quality}.d{max_dimension if max_dimension is not None else 0}.webp"
+        cache_path  = os.path.join(self.app.convcachepath, cache_name)
+        return self.__cached_conversion(media_path, cache_path, quality, max_dimension)
+
+    def __cached_conversion(self, media_path, cache_path, quality, max_dimension):
+        with self.conversion_cache_lock:
+            if os.path.isfile(cache_path):
+                RNS.log(f"Using cached media conversion for: {media_path}", RNS.LOG_VERBOSE)
+                try: os.utime(cache_path, None)
+                except Exception: pass
+                return cache_path
+
+            try:
+                os.makedirs(self.app.convcachepath, exist_ok=True)
+                converted_path = convert_file_to_webp(media_path, quality=quality, max_dimension=max_dimension)
+                if converted_path is False: return False
+
+                shutil.move(converted_path, cache_path)
+                self._prune_conversion_cache()
+                RNS.log(f"Converted media: {media_path} -> {cache_path}", RNS.LOG_VERBOSE)
+                return cache_path
+
+            except Exception as e:
+                RNS.log(f"Error during media conversion of {media_path}: {e}", RNS.LOG_WARNING)
+                return False
+
+    def _prune_conversion_cache(self):
+        try:
+            entries = []
+            for name in os.listdir(self.app.convcachepath):
+                path = os.path.join(self.app.convcachepath, name)
+                try:
+                    st = os.stat(path)
+                    if st.st_size > 0:
+                        entries.append((st.st_mtime, st.st_size, path))
+                except OSError: pass
+
+            total = sum(e[1] for e in entries)
+            entries.sort()
+            while len(entries) > Node.CONVERSION_CACHE_MAX_FILES or total > Node.CONVERSION_CACHE_MAX_BYTES:
+                _, size, path = entries.pop(0)
+                total -= size
+                try: os.unlink(path)
+                except OSError: pass
+        except OSError: pass
+
     def request_allowed(self, file_path, remote_identity):
         if file_path.lower().endswith(".allowed"): return False
         allowed_path = file_path+".allowed"
@@ -235,7 +313,6 @@ class Node:
         RNS.log("Denying request, remote identity was not in list of allowed identities", RNS.LOG_VERBOSE)
         return False
 
-    # TODO: Improve file handling, this will be slow for large files
     def serve_file(self, path, data, request_id, remote_identity, requested_at):
         RNS.log("File request "+RNS.prettyhexrep(request_id)+" for: "+str(path), RNS.LOG_VERBOSE)
         try:

nomadnet/NomadNetworkApp.py

@@ -112,7 +112,9 @@ class NomadNetworkApp:
         self.storagepath       = self.configdir+"/storage"
         self.identitypath      = self.configdir+"/storage/identity"
         self.cachepath         = self.configdir+"/storage/cache"
-        self.imagecachepath    = self.configdir+"/storage/cache/images"
+        self.imagecachepath    = self.configdir+"/storage/cache/images/browser"
+        self.convcachepath     = self.configdir+"/storage/cache/images/converted_node"
+        self.dispcachepath     = self.configdir+"/storage/cache/images/converted_disp"
         self.resourcepath      = self.configdir+"/storage/resources"
         self.conversationpath  = self.configdir+"/storage/conversations"
         self.directorypath     = self.configdir+"/storage/directory"
@@ -179,6 +181,12 @@ class NomadNetworkApp:
         if not os.path.isdir(self.imagecachepath):
             os.makedirs(self.imagecachepath)
 
+        if not os.path.isdir(self.convcachepath):
+            os.makedirs(self.convcachepath)
+
+        if not os.path.isdir(self.dispcachepath):
+            os.makedirs(self.dispcachepath)
+
         if not os.path.isdir(self.resourcepath):
             os.makedirs(self.resourcepath)
 
@@ -308,6 +316,7 @@ class NomadNetworkApp:
                     "last_announce": None,
                     "node_last_announce": None,
                     "propagation_node": None,
+                    "quit_confirm": True,
                     "last_lxmf_sync": 0,
                     "node_connects": 0,
                     "served_page_requests": 0,

nomadnet/RRC.py

@@ -118,7 +118,7 @@ def _msg_id():
 # greedy .+ intentionally captures nicks containing parens like "user (alt) (deadbeefcafe)"
 _WHO_ENTRY_RE = re.compile(
     r"(?:^|,\s)"
-    r"(?:(?P<bh>[0-9a-fA-F]{32})|(?P<nick>.+?)\s\((?P<np>[0-9a-fA-F]{12})\))"
+    r"(?:(?P<bh>[0-9a-fA-F]{64})|(?P<nick>.+?)\s\((?P<np>[0-9a-fA-F]{12})\))"
     r"(?=,\s|$)"
 )
 
@@ -170,6 +170,16 @@ def _parse_room_list_notice(text):
     return rooms
 
 
+# On low-MTU links the hub may deliver a /list reply as a sequence of
+# chunked NOTICE packets (one line per packet) instead of a single notice
+# or a resource transfer. The chunks carry no continuation marker, but each
+# line has a recognisable shape. Lines matching it, arriving within a short
+# window after a /list header, are treated as continuation fragments and
+# merged into the room list instead of being recorded as chat messages.
+_LIST_FRAGMENT_RE = re.compile(r"^  (?P<name>\S+?)(?: - (?P<topic>.*))?$")
+LIST_FRAGMENT_WINDOW = 60.0
+
+
 def _make_envelope(msg_type, src, room=None, body=None, nick=None, mid=None, ts=None):
     env = {
         K_V:   RRC_VERSION,
@@ -235,6 +245,11 @@ class RRCHub:
         self.mention_rooms = set()
         self.members = {}
         self.nicks = {}
+        # Display-only member entries derived from 12-hex identity prefixes
+        # in /who replies. Populated when the full identity hash of a nicked
+        # member is not yet known, and promoted to regular members as soon
+        # as the full hash is learned from room traffic or join fanouts.
+        self.prefix_members = {}
 
         self.auto_reconnect = False
         self.auto_list = False
@@ -259,6 +274,7 @@ class RRCHub:
         self.available_rooms = {}
         self._silent_list_pending = 0
         self._silent_who_rooms = set()
+        self._list_fragment_until = 0.0
 
         self.nick_override = None
         self._pending_joins = set()
@@ -290,6 +306,7 @@ class RRCHub:
             self.unread_rooms.discard(r)
             self.mention_rooms.discard(r)
             self.members.pop(r, None)
+            self.prefix_members.pop(r, None)
         self._delete_history(r)
         self.manager.save()
         self.manager._notify_change(self)
@@ -306,7 +323,23 @@ class RRCHub:
 
     def get_members(self, room):
         with self._lock:
-            return list(self.members.get(room, set()))
+            return list(self.members.get(room, set()) | self.prefix_members.get(room, set()))
+
+    def _upgrade_prefix_member(self, room, full_hash):
+        # Once a full identity hash is learned from room traffic or join
+        # fanouts, promote any display-only prefix member that matches it,
+        # so the entry becomes a full, addressable member. The prefix
+        # entry's nickname is carried over unless the promoting source
+        # provides one.
+        pset = self.prefix_members.get(room)
+        if pset:
+            for p in list(pset):
+                if full_hash.startswith(p):
+                    pset.discard(p)
+                    nick = self.nicks.pop(p, None)
+                    if nick and full_hash not in self.nicks:
+                        self.nicks[full_hash] = nick
+                    break
 
     def display_name_for(self, peer):
         if not isinstance(peer, (bytes, bytearray)):
@@ -467,6 +500,7 @@ class RRCHub:
             self.welcomed = False
             self.motd = None
             self.members.clear()
+            self.prefix_members.clear()
             self._resource_expectations.clear()
             self._pending_joins.clear()
             self._pending_parts.clear()
@@ -844,23 +878,50 @@ class RRCHub:
     def _process_notice_text(self, text):
         # Parse hub service notices (/list and /who replies) regardless of
         # whether they arrived as a packet or a resource transfer. Returns
-        # True when the notice was consumed silently (auto /list or /who)
+        # True when the notice was consumed silently (auto /list or /who,
-        # and should not be recorded to the message log.
+        # or a /list continuation fragment) and should not be recorded to
+        # the message log.
         parsed = _parse_room_list_notice(text)
         if parsed is not None:
             with self._lock:
                 self.available_rooms = {strip_modifiers(name): strip_modifiers(topic) for name, topic in parsed.items()}
+                # The first line of a /list reply opens a short window in
+                # which continuation fragments (chunked NOTICE packets on
+                # low-MTU links) are absorbed into the room list.
+                self._list_fragment_until = time.monotonic()+LIST_FRAGMENT_WINDOW
                 silent = self._silent_list_pending > 0
                 if silent:
                     self._silent_list_pending -= 1
             self.manager._notify_change(self)
-            if silent:
+            if silent: return True
+        with self._lock:
+            list_window_active = time.monotonic() < self._list_fragment_until
+        if list_window_active:
+            fm = _LIST_FRAGMENT_RE.match(text)
+            if fm is not None:
+                with self._lock:
+                    frag_name = fm.group("name").strip().lower()
+                    frag_topic = fm.group("topic")
+                    if frag_topic:
+                        frag_topic = strip_modifiers(frag_topic.strip()) or None
+                    else:
+                        frag_topic = None
+                    self.available_rooms[frag_name] = frag_topic
+                    self._list_fragment_until = time.monotonic()+LIST_FRAGMENT_WINDOW
+                self.manager._notify_change(self)
                 return True
         parsed_who = _parse_who_notice(text)
         if parsed_who is not None:
             who_room, who_entries = parsed_who
             with self._lock:
                 members = self.members.setdefault(who_room, set())
+                # Treat /who replies as fresh roster snapshots: replace any
+                # display-only prefix members from earlier replies before
+                # applying this one.
+                prefix_members = self.prefix_members.setdefault(who_room, set())
+                for p in list(prefix_members):
+                    self.nicks.pop(p, None)
+                prefix_members.clear()
                 for nick, hash_hex in who_entries:
                     try:
                         hash_bytes = bytes.fromhex(hash_hex)
@@ -868,11 +929,23 @@ class RRCHub:
                         continue
                     if nick is None:
                         members.add(hash_bytes)
+                        self._upgrade_prefix_member(who_room, hash_bytes)
                         continue
+                    known = None
                     for ph in members:
                         if ph.startswith(hash_bytes):
-                            self.nicks[ph] = nick
+                            known = ph
                             break
+                    if known is not None:
+                        self.nicks[known] = nick
+                    else:
+                        # A /who line only carries a 12-hex identity prefix
+                        # for nicked members. If the full hash is not yet
+                        # known, keep the prefix as a display-only member.
+                        # It is promoted to a full member as soon as the
+                        # hash is learned from room traffic or fanouts.
+                        prefix_members.add(hash_bytes)
+                        self.nicks[hash_bytes] = nick
                 silent_who = who_room in self._silent_who_rooms
                 if silent_who:
                     self._silent_who_rooms.discard(who_room)
@@ -987,8 +1060,10 @@ class RRCHub:
                     members = self.members.setdefault(r, set())
                     for h in body_hashes:
                         members.add(h)
+                        self._upgrade_prefix_member(r, h)
                     if own_hash is not None:
                         members.add(own_hash)
+                        self._upgrade_prefix_member(r, own_hash)
 
                     # rrcd 0.3.2 attaches advisory K_NICK to fanout JOINED; learn it
                     # so display_name_for() can render the nick instead of a hash prefix.
@@ -1043,12 +1118,19 @@ class RRCHub:
                             self.nicks[ph] = parter_nick
 
                     members = self.members.get(r)
+                    pset = self.prefix_members.get(r)
                     if members is not None:
                         for h in body_hashes:
                             members.discard(h)
+                            if pset:
+                                for p in list(pset):
+                                    if h.startswith(p):
+                                        pset.discard(p)
+                                        self.nicks.pop(p, None)
                     if self_part:
                         self.rooms.discard(r)
                         self.members.pop(r, None)
+                        self.prefix_members.pop(r, None)
 
                 if self_part:
                     self.manager.save()
@@ -1075,7 +1157,9 @@ class RRCHub:
                 with self._lock:
                     self.nicks[bytes(src)] = nick
                     if isinstance(room, str) and room:
-                        self.members.setdefault(room.strip().lower(), set()).add(bytes(src))
+                        rn = room.strip().lower()
+                        self.members.setdefault(rn, set()).add(bytes(src))
+                        self._upgrade_prefix_member(rn, bytes(src))
             if isinstance(body, str):
                 msg = RRCMessage(
                     "msg",
@@ -1108,7 +1192,9 @@ class RRCHub:
                 with self._lock:
                     self.nicks[bytes(src)] = nick
                     if isinstance(room, str) and room:
-                        self.members.setdefault(room.strip().lower(), set()).add(bytes(src))
+                        rn = room.strip().lower()
+                        self.members.setdefault(rn, set()).add(bytes(src))
+                        self._upgrade_prefix_member(rn, bytes(src))
             if isinstance(body, str):
                 msg = RRCMessage(
                     "action",
@@ -1132,12 +1218,47 @@ class RRCHub:
             room = env.get(K_ROOM)
             src  = env.get(K_SRC)
             if isinstance(body, str):
-                if self._process_notice_text(body):
+                #################### HACK ZONE #####################
-                    return
+                # This is the kind of crazy mess that happens when
+                # an LLM designs a chat protocol, and the client
+                # maintainer is too open-minded, decides to go with
+                # it, and now is responsible for the fundamental
+                # architectural chaos.
+                #
+                # Yes, I love RRC, but trying to figure out an
+                # elegant way to solve the underlying issues that
+                # make this kind of mess necessary revealed the
+                # following answer: There is none.
+                #
+                # There are so many bugs, weird cases, architectural
+                # mismatches, impossible situations and general
+                # weirdness here, that I am just not going to go and
+                # attempt to sort it all out. Sorry, I don't have the
+                # time, so a hack like this is what it will be. This
+                # makes things actually work semi-expectedly, even
+                # over low-bandwidth links and such, so hacky fix it
+                # is. Until we have something better, this will do
+                # just fine. On a related note, I was considering
+                # taking up maintenance of rrcd, now that it seems to
+                # not really be worked on or updated anymore, but
+                # looking at the entire system, I think it's better
+                # left as it; it works fine, albeit with a few quirks
+                # for it's purpose. My future work in chat-related
+                # protocols on Reticulum will be focused on a better
+                # architecture, incorporating everything we learned
+                # from RRC, and making it Reticulum-native from the
+                # start, instead of mirroring IRC too closely.
+                #  - Mark
+                was_parsed = self._process_notice_text(body)
                 room_n = room.strip().lower() if isinstance(room, str) else None
                 if room_n is None and isinstance(body, str) and body.strip():
-                    with self._lock:
+                    probably_motd = True
-                        self.motd = strip_modifiers(body)
+                    if was_parsed: probably_motd = False
+                    if body.lstrip().lower().startswith("registered public rooms:"): probably_motd = False; was_parsed = True
+                    if body.lstrip().lower().startswith("members in "): probably_motd = False; was_parsed = True
+                    if probably_motd:
+                        with self._lock: self.motd = strip_modifiers(body)
+                        was_parsed = False
                     self.manager._notify_change(self)
                 msg = RRCMessage(
                     "notice",
@@ -1147,7 +1268,7 @@ class RRCHub:
                     body,
                     _now_ms(),
                 )
-                self._record_notice(msg)
+                if not was_parsed: self._record_notice(msg)
             return
 
         if t == T_ERROR:

nomadnet/_version.py

@@ -1 +1 @@
-__version__ = "1.4.1"
+__version__ = "1.4.4"

nomadnet/ui/textui/Browser.py

@@ -3,6 +3,7 @@ import LXMF
 import io
 import os
 import time
+import bisect
 import urwid
 import shutil
 import nomadnet
@@ -52,11 +53,11 @@ class BrowserFrame(urwid.Frame):
         elif self.focus_position == "body":
             if key == "down" or key == "up":
                 try:
-                    if hasattr(self.delegate, "page_pile") and self.delegate.page_pile:
+                    if hasattr(self.delegate, "page_list") and self.delegate.page_list:
                         def df(loop, user_data):
                             st = None
-                            if self.delegate.page_pile:
+                            if self.delegate.page_list:
-                                nf = self.delegate.page_pile.focus
+                                nf = self.delegate.page_list.focus
                                 if hasattr(nf, "key_timeout"):
                                     st = nf
                                 elif hasattr(nf, "original_widget"):
@@ -80,6 +81,168 @@ class BrowserFrame(urwid.Frame):
         else:
             return super(BrowserFrame, self).keypress(size, key)
 
+class PageList(urwid.ListBox):
+    def __init__(self, body):
+        super().__init__(body)
+        self._page_size = (0, 0)
+        self._page_focus = None
+        self._page_layouts = {}
+
+    def invalidate_layout(self):
+        self._page_layouts = {}
+        self._invalidate()
+
+    def _layout(self, maxcol):
+        offsets = self._page_layouts.get(maxcol)
+        if offsets is None or len(offsets) != len(self.body) + 1:
+            layout_started = time.time()
+            offsets = [0]
+            for widget in self.body:
+                offsets.append(offsets[-1] + widget.rows((maxcol,), True))
+            self._page_layouts[maxcol] = offsets
+            # RNS.log(f" rebuilt {len(self.body)} widgets, {offsets[-1]} rows, width {maxcol}, {(time.time()-layout_started)*1000:.1f}ms", RNS.LOG_DEBUG)
+        return offsets
+
+    def calculate_visible(self, size, focus=False):
+        visible = super().calculate_visible(size, focus)
+        if visible is not None and self.body:
+            self._sync_layout(size[0], visible)
+        return visible
+
+    def _sync_layout(self, maxcol, visible):
+        # Validate the cached layout against the live geometry of the
+        # currently visible widgets, and delta-update the cached offsets when a
+        # visible widget has changed height since the layout was built without
+        # rescanning the entire document on every render.
+
+        if not self.body or not self._page_layouts: return
+
+        offsets = self._page_layouts.get(maxcol)
+        if offsets is None or len(offsets) != len(self.body) + 1: return
+
+        def sync_position(pos, live_rows):
+            if pos < 0 or pos >= len(self.body) or live_rows < 0: return
+            cached_rows = offsets[pos + 1] - offsets[pos]
+            if cached_rows == live_rows: return
+
+            delta = live_rows - cached_rows
+            for i in range(pos + 1, len(offsets)): offsets[i] += delta
+
+            # Other cached widths may also be affected by this change. Sync
+            # them using the changed widget's live rows at their widths.
+            for width, other in list(self._page_layouts.items()):
+                if width == maxcol or len(other) != len(offsets): continue
+                other_cached = other[pos + 1] - other[pos]
+                other_live = self.body[pos].rows((width,), True)
+                if other_cached == other_live: continue
+                other_delta = other_live - other_cached
+                for i in range(pos + 1, len(other)): other[i] += other_delta
+
+        middle = visible.middle
+        sync_position(middle.focus_pos, middle.focus_rows)
+        for item in visible.top.fill:    sync_position(item.position, item.rows)
+        for item in visible.bottom.fill: sync_position(item.position, item.rows)
+
+    def render(self, size, focus=False):
+        self._page_size = size
+        if self.body and self.focus is not self._page_focus:
+            self._page_focus = self.focus
+            widget = self.focus
+            while widget is not None:
+                widget._invalidate()
+                widget = getattr(widget, "original_widget", None)
+        return super().render(size, focus)
+
+    def rows_max(self, size=None, focus=False):
+        size = size or self._page_size
+        return self._layout(size[0])[-1]
+
+    def get_scrollpos(self, size=None, focus=False):
+        size = size or self._page_size
+        if not self.body or not size[1]: return 0
+
+        middle, top, bottom = self.calculate_visible(size, focus)
+        if middle is None: return 0
+        return self._layout(size[0])[self.top_position(size)] + top.trim
+
+    def set_scrollpos(self, position):
+        size = self._page_size
+        maxcol,  maxrow = size
+        if not self.body or not maxrow: return
+
+        offsets = self._layout(maxcol)
+        position = max(0, min(int(position), max(0, offsets[-1] - maxrow)))
+        pos = bisect.bisect_left(offsets, position, 0, len(self.body))
+        if pos == len(self.body) or offsets[pos] - position >= maxrow: pos -= 1
+
+        # Guard against the cached layout being inconsistent
+        # with the live widget geometry. Without it, change_focus
+        # can be called with an offset that the live target widget
+        # cannot satisfy, raising ListBoxError.
+        pos = max(0, min(pos, len(self.body) - 1))
+        offset_inset = offsets[pos] - position
+        if offset_inset > maxrow - 1: offset_inset = maxrow - 1
+        elif offset_inset < 0:
+            tgt_rows = max(1, self.body[pos].rows((maxcol,), True))
+            if offset_inset < 1 - tgt_rows: offset_inset = 1 - tgt_rows
+
+        self.change_focus(size, pos, offset_inset)
+
+    def top_position(self, size=None):
+        size = size or self._page_size
+        if not self.body or not size[1]: return 0
+        middle, top, bottom = self.calculate_visible(size, True)
+        if middle is None: return 0
+        return top.fill[-1].position if top.fill else middle.focus_pos
+
+    def scroll(self, rows):
+        self.set_scrollpos(self.get_scrollpos() + rows)
+
+    def focused_widget_name(self):
+        widget = self.focus
+        while getattr(widget, "original_widget", None) is not None: widget = widget.original_widget
+        return type(widget).__name__
+
+    def keypress(self, size, key):
+        self._page_size = size
+        key_started   = time.time()
+        focus_before  = self.focus_position if self.body else None
+        scroll_before = self.get_scrollpos(size, True)
+        unhandled_key = self.handle_key(size, key)
+        
+        # RNS.log(f"  {key}: {unhandled_key} focus {focus_before} -> {self.focus_position if self.body else None} ({self.focused_widget_name()}) scrollpos {scroll_before}->{self.get_scrollpos(size, True)}, {(time.time()-key_started) * 1000:.1f}ms", RNS.LOG_DEBUG)
+        return unhandled_key
+
+    def handle_key(self, size, key):
+        try:
+            if key in ("page up", "page down", "home", "end"):
+                focus_widget = self.focus
+                if focus_widget is not None and focus_widget.selectable():
+                    key = focus_widget.keypress((size[0],), key)
+                    if key is None:
+                        self.make_cursor_visible(size)
+                        return None
+
+                if   key == "page down": self.scroll(size[1] - 1)
+                elif key == "page up":   self.scroll(1 - size[1])
+                elif key == "home":      self.set_scrollpos(0)
+                elif key == "end":       self.set_scrollpos(self.rows_max(size))
+                return None
+
+            unhandled_key = super().keypress(size, key)
+            if key in ("up", "down"): return None
+            return unhandled_key
+
+        except Exception as e:
+            RNS.log(f"Error while handling browser keypress: {e}", RNS.LOG_ERROR)
+            RNS.trace_exception(e)
+            return None
+
+    def mouse_event(self, size, event, button, col, row, focus):
+        self._page_size = size
+        if button in (4, 5): return False
+        return super().mouse_event(size, event, button, col, row, focus)
+
 class Browser:
     DEFAULT_PATH       = "/page/index.mu"
     DEFAULT_TIMEOUT    = 10
@@ -133,13 +296,15 @@ class Browser:
         self.link_target = None
         self.frame = None
         self.attr_maps = []
-        self.page_pile = None
+        self.page_list = None
+        self.page_rows = []
         self.page_partials = {}
         self.page_images = {}
         self.updater_running = False
         self.image_updater_running = False
         self.partial_updater_lock = Lock()
         self.image_updater_lock = Lock()
+        self.image_cache_lock = Lock()
         self.image_mode = self.app.config["textui"]["image_loading"]
         self.image_rendering_supported = _termlib.is_kitty_supported() and self.image_mode != IMGLOAD_NEVER
         self.build_display()
@@ -163,27 +328,22 @@ class Browser:
         else:                              RNS.log(f"In-browser image rendering is not supported", RNS.LOG_DEBUG)
 
     def current_url(self):
-        if self.destination_hash == None:
+        if self.destination_hash == None: return ""
-            return ""
         path = "" if self.path == None else self.path
         url = RNS.hexrep(self.destination_hash, delimit=False)+":"+path
 
         if isinstance(self.request_data, dict) and self.request_data:
             parts = []
             for k, v in self.request_data.items():
-                if not isinstance(k, str):
+                if not isinstance(k, str):   continue
-                    continue
+                if not k.startswith("var_"): continue
-                if not k.startswith("var_"):
-                    continue
                 parts.append(k[4:]+"="+str(v))
-            if parts:
+            if parts: url += "`" + "|".join(parts)
-                url += "`" + "|".join(parts)
 
         return url
 
     def url_hash(self, url):
-        if url == None:
+        if url == None: return None
-            return None
         else:
             url = url.encode("utf-8")
             return RNS.hexrep(RNS.Identity.full_hash(url), delimit=False)
@@ -344,8 +504,6 @@ class Browser:
         anchors     = getattr(self.attr_maps, "anchors", None) or {}
         header_rows = getattr(self.attr_maps, "header_rows", None) or []
 
-        cols = self._content_cols()
-
         target_idx = None
         if name:
             target_idx = anchors.get(name)
@@ -356,11 +514,11 @@ class Browser:
         else:
             current = 0
             try:
-                current = self.browser_body.original_widget.original_widget.get_scrollpos()
+                current = self.page_rows[self.page_list.top_position()]
             except Exception:
                 current = 0
             for hr in header_rows:
-                if self._rows_above(hr, cols) > current:
+                if hr > current:
                     target_idx = hr
                     break
             if target_idx is None:
@@ -368,12 +526,9 @@ class Browser:
 
         self.reveal_index(int(target_idx))
 
-        row_offset = self._rows_above(int(target_idx), cols)
-
         try:
-            scrollable = self.browser_body.original_widget.original_widget
+            self.page_list.set_focus(self.page_rows.index(int(target_idx)))
-            scrollable.anchor_cursor_update = True
+            self.page_list.set_focus_valign("top")
-            scrollable.set_scrollpos(row_offset)
 
         except Exception as e: RNS.log("Anchor jump failed: "+str(e), RNS.LOG_ERROR)
 
@@ -391,18 +546,6 @@ class Browser:
 
         return max(40, cols)
 
-    def _rows_above(self, index, cols):
-        if index <= 0 or not self.attr_maps: return 0
-
-        hidden = self.hidden_indices()
-        total = 0
-        for i in range(min(index, len(self.attr_maps))):
-            if i in hidden: continue
-            try: total += self.attr_maps[i].rows((cols,))
-            except Exception: total += 1
-
-        return total
-
     def handle_lxmf_link(self, link_target):
         try:
             def san(name):
@@ -497,7 +640,8 @@ class Browser:
         self.browser_header = urwid.Text("")
         self.browser_footer = urwid.Text("")
 
-        self.page_pile = None
+        self.page_list = None
+        self.page_rows = []
         self.page_partials = {}
         self.page_images = {}
         self.browser_body = urwid.Filler(
@@ -573,7 +717,8 @@ class Browser:
     def update_display(self):
         if self.status == Browser.DISCONECTED:
             self.display_widget.set_attr_map({None: "browser_inactive"})
-            self.page_pile = None
+            self.page_list = None
+            self.page_rows = []
             self.page_partials = {}
             self.page_images = {}
             self.browser_body = urwid.Filler(
@@ -647,21 +792,25 @@ class Browser:
 
     def update_page_display(self):
         self._purge_page_images()
-        pile = urwid.Pile(self.attr_maps)
+        self.page_rows = list(range(len(self.attr_maps)))
-        pile.automove_cursor_on_scroll = True
+        self.page_list = PageList(urwid.SimpleFocusListWalker(list(self.attr_maps)))
-        self.page_pile = pile
         self.page_partials = {}
         self.page_images = {}
-        self.browser_body = urwid.AttrMap(ScrollBar(Scrollable(pile, force_forward_keypress=True), thumb_char="\u2503", trough_char=" "), "scrollbar")
+        self.browser_body = urwid.AttrMap(ScrollBar(self.page_list, thumb_char="\u2503", trough_char=" "), "scrollbar")
         self.detect_partials()
         if self.should_load_images(): self.detect_images()
         self.init_folds()
 
+    def refresh_page_layout(self):
+        if self.page_list is not None: self.page_list.invalidate_layout()
+
     def image_cache_path(self, url):
+        destination_hash, path = self.parse_url(url)
+        url = f"{RNS.hexrep(destination_hash, delimit=False)}:{path}"
         url_hash = self.url_hash(url)
         if not url_hash: return None
         else:
-            path = f"{self.app.cachepath}/images/{url_hash}"
+            path = f"{self.app.imagecachepath}/{url_hash}"
             return path
 
     def resolve_image(self, url):
@@ -723,6 +872,7 @@ class Browser:
 
                     self.page_images[o.image_id] = image
 
+        self.refresh_page_layout()
         if   force_reload: self.reload()
         elif len(self.page_images) > 0: self.start_image_updater()
 
@@ -767,7 +917,16 @@ class Browser:
         if image_destination_hash == self.loopback:
             local_image = path.replace("/media/", "")
             local_path = f"{self.app.pagespath}/{local_image}"
-            if self.image_rendering_supported: w.load(local_path)
+
+            if self.app.node:
+                image_ext = os.path.splitext(local_path)[-1]
+                if not image_ext in self.app.node.NATIVE_MEDIA_EXTS:
+                    RNS.log(f"Non-native media type for {local_path}, attempting auto-conversion", RNS.LOG_DEBUG)
+                    local_path = self.app.node.convert_media_to_webp(local_path)
+
+            if self.image_rendering_supported:
+                w.load(local_path)
+                self.refresh_page_layout()
             image["updated"] = time.time()
             return
 
@@ -906,14 +1065,44 @@ class Browser:
                             file_handle = request_receipt.response
                             file_destination = cachepath
                             shutil.move(file_handle.name, file_destination)
+                            threading.Thread(target=self.clean_image_cache, daemon=True).start()
 
                     resolved_path = self.resolve_image(url)
-                    if resolved_path and self.image_rendering_supported: w.load(resolved_path, remote_source=True)
+                    if resolved_path and self.image_rendering_supported:
+                        w.load(resolved_path, remote_source=True)
+                        self.refresh_page_layout()
 
                 except Exception as e:
                     RNS.log("Error while handling image response: "+str(e), RNS.LOG_ERROR)
                     image["failed"] = True
 
+    IMAGE_CACHE_MAX_FILES = 256
+    IMAGE_CACHE_MAX_BYTES = 192*1024*1024
+    def clean_image_cache(self):
+        if self.image_cache_lock.locked(): return
+        with self.image_cache_lock:
+            try:
+                entries = []
+                for name in os.listdir(self.app.imagecachepath):
+                    path = os.path.join(self.app.imagecachepath, name)
+                    try:
+                        st = os.stat(path)
+                        if st.st_size > 0: entries.append((st.st_mtime, st.st_size, path))
+                    except OSError as exc: RNS.log(f"Could not get cached file information for {path}: {e}", RNS.LOG_DEBUG)
+
+                cleaned_b = 0; cleaned_c = 0
+                total = sum(e[1] for e in entries)
+                entries.sort()
+                while len(entries) > self.IMAGE_CACHE_MAX_FILES or total > self.IMAGE_CACHE_MAX_BYTES:
+                    _, size, path = entries.pop(0)
+                    total -= size
+                    cleaned_b += size
+                    cleaned_c += 1
+                    try: os.unlink(path)
+                    except OSError as exc: RNS.log(f"Could not remove cached file {path}: {exc}")
+                if cleaned_b or cleaned_c: RNS.log(f"Cleaned {RNS.prettysize(cleaned_b)} / {cleaned_c} image{'s' if cleaned_c != 1 else ''} from cache", RNS.LOG_DEBUG)
+            except Exception as exc: RNS.log(f"Error while cleaning browser image cache: {exc}", RNS.LOG_ERROR)
+
     # Render an arbitrary markup buffer (used by the page editor preview) and
     # return the content widget, with partials/folds/fields wired as usual
     def render_markup_buffer(self, markup):
@@ -979,15 +1168,14 @@ class Browser:
         return hidden
 
     def rebuild_visible(self):
-        pile = getattr(self, "page_pile", None)
+        page = getattr(self, "page_list", None)
-        if pile is None: return
+        if page is None: return
         hidden = self.hidden_indices()
-        opts = pile.options()
+        self.page_rows = [i for i in range(len(self.attr_maps)) if i not in hidden]
-        pile.contents = [(w, opts) for i, w in enumerate(self.attr_maps) if i not in hidden]
+        focus = page.focus_position if page.body else 0
-        try:
+        page.body[:] = [self.attr_maps[i] for i in self.page_rows]
-            if pile.focus_position >= len(pile.contents):
+        page.invalidate_layout()
-                pile.focus_position = max(0, len(pile.contents)-1)
+        if page.body: page.set_focus(min(focus, len(page.body)-1))
-        except Exception: pass
 
     def fold_changed(self, heading_widget):
         if getattr(heading_widget, "collapsed", False):
@@ -996,9 +1184,9 @@ class Browser:
             self.collapsed_headings.discard(heading_widget)
         self.rebuild_visible()
         try:
-            for pos, (w, _) in enumerate(self.page_pile.contents):
+            for pos, w in enumerate(self.page_list.body):
                 if getattr(w, "_original_widget", None) is heading_widget:
-                    self.page_pile.focus_position = pos
+                    self.page_list.set_focus(pos)
                     break
         except Exception: pass
 
@@ -1067,6 +1255,7 @@ class Browser:
                     url = partial["url"]
                     pile = partial["pile"]
                     pile.contents = [(urwid.Text(f"Could not load partial {url}: The resource transfer failed"), pile.options())]
+                    self.refresh_page_layout()
                 except Exception as e:
                     RNS.log(f"Error in partial failed callback: {e}", RNS.LOG_ERROR)
                     RNS.trace_exception(e)
@@ -1085,6 +1274,7 @@ class Browser:
                     partial["attr_maps"] = markup_to_attrmaps(strip_modifiers(partial["content"]), url_delegate=self, fg_color=self.page_foreground_color, bg_color=self.page_background_color)
                     pile = partial["pile"]
                     pile.contents = [(e, pile.options()) for e in partial["attr_maps"]]
+                    self.refresh_page_layout()
 
                 except Exception as e:
                     RNS.trace_exception(e)
@@ -1098,6 +1288,7 @@ class Browser:
             pile = partial["pile"]
             url = partial["url"]
             pile.contents = [(urwid.Text(f"Could not load partial {url}: {e}"), pile.options())]
+            self.refresh_page_layout()
             return
 
         if partial_destination_hash != self.loopback and not RNS.Transport.has_path(partial_destination_hash):

nomadnet/ui/textui/Editor.py

@@ -445,6 +445,8 @@ class PageEditorDisplay():
         self.register_styles()
 
         self.preview_browser = Browser(self.app, "nomadnetwork", "node", delegate=None)
+        self.preview_browser.loopback = self.app.node.destination.hash
+        self.preview_browser.destination_hash = self.app.node.destination.hash
         self.preview_browser.handle_link = self._preview_handle_link
         self.preview_browser.marked_link = lambda *a, **k: None
         self.preview_browser._content_cols = self._preview_cols

nomadnet/ui/textui/MicronParser.py

@@ -603,14 +603,14 @@ def parse_line(line, state, url_delegate):
                 if len(line) == 2:
                     divider_char = line[1]
                     # Control characters don't make sense here and otherwise crash nomadnet
-                    if ord(divider_char) < 32:
+                    if ord(divider_char) < 32: divider_char = "\u2500"
-                        divider_char = "\u2500"
+                    # Check if the divider char is renderable
-                else:
+                    try: urwid.Divider(divider_char).render((30,))
-                    divider_char = "\u2500"
+                    except Exception as e: divider_char = "\u2500"
-                if state["depth"] == 0:
+                else: divider_char = "\u2500"
-                    return [urwid.Divider(divider_char)]
+
-                else:
+                if state["depth"] == 0: return [urwid.Divider(divider_char)]
-                    return [urwid.Padding(urwid.Divider(divider_char), left=left_indent(state), right=right_indent(state))]
+                else:                   return [urwid.Padding(urwid.Divider(divider_char), left=left_indent(state), right=right_indent(state))]
 
         output = make_output(state, line, url_delegate, pre_escape)
 
@@ -1159,6 +1159,8 @@ class LinkableText(urwid.Text):
         super().__init__(text, align=align)
         self.delegate = delegate
         self._cursor_position = 0
+        self._cursor_promised = None
+        self._cursor_promised_at = 0
         self.key_timeout = 2
         self.in_columns = False
         if self.delegate != None:
@@ -1265,28 +1267,35 @@ class LinkableText(urwid.Text):
     def kt_event(self, loop, user_data):
         self._invalidate()
 
+    def cursor_visible(self):
+        return self.delegate == None or time.time() < self.delegate.last_keypress+self.key_timeout
+
     def render(self, size, focus=False):
-        now = time.time()
         c = super().render(size, focus)
-
+        if focus:
-        if focus and (self.delegate == None or now < self.delegate.last_keypress+self.key_timeout):
+            if time.time() < self._cursor_promised_at+0.5:
-            c = urwid.CompositeCanvas(c)
+                coords = self._cursor_promised
-            c.cursor = self.get_cursor_coords(size)
+            else:
-            if self.delegate != None:
+                coords = self.get_cursor_coords(size)
-                self.peek_link()
+            if coords != None:
-
+                c = urwid.CompositeCanvas(c)
+                c.cursor = coords
+                if self.delegate != None:
+                    self.peek_link()
         return c
 
     def get_cursor_coords(self, size):
-            if self._cursor_position > len(self.text):
+        self._cursor_promised_at = time.time()
-                return None
+        self._cursor_promised = None
-
+        if not self.cursor_visible() or self._cursor_position > len(self.text):
-            (maxcol,) = size
+            return None
-            trans = self.get_line_translation(maxcol)
+        (maxcol,) = size
-            x, y = calc_coords(self.text, trans, self._cursor_position)
+        trans = self.get_line_translation(maxcol)
-            if maxcol <= x:
+        x, y = calc_coords(self.text, trans, self._cursor_position)
-                return None
+        if maxcol <= x:
-            return x, y
+            return None
+        self._cursor_promised = (x, y)
+        return x, y
 
     def mouse_event(self, size, event, button, x, y, focus):
         try:

nomadnet/ui/textui/images/_termlib.py

@@ -10,9 +10,11 @@ import os
 import select
 import shutil
 import sys
-import termios
 import time
 
+try: import termios
+except: termios = None
+
 from . import _ctlseqs
 
 QUERY_TIMEOUT = 0.3
@@ -43,7 +45,7 @@ def _open_tty():
 def query_tty(request, timeout=QUERY_TIMEOUT):
     # Sends a query to the terminal and returns the raw response.
     global _query_failed
-    if _query_failed: return b""
+    if _query_failed or not termios: return b""
 
     fd = _open_tty()
     if fd == -1:
@@ -121,6 +123,7 @@ def _cell_size_from_winsize(buf):
 
 def _query_cell_size():
     # Determines the current cell size in pixels, or None if unknown.
+    if not termios: return None
     fd = _open_tty()
     if fd != -1:
         # First, try the TIOCGWINSZ pixel fields (fast, no query)

nomadnet/ui/textui/images/_webp.py

@@ -25,7 +25,6 @@
 # Environment overrides:
 # 
 #     NOMADNET_IMAGE_BACKEND   force a specific backend (strict)
-#     NOMADNET_IMAGE_CACHE     override the conversion cache directory
 
 import ctypes
 import ctypes.util
@@ -115,12 +114,13 @@ def _encode_png_rgba(rgba, width, height):
 # Conversion cache #
 ####################
 
+CONV_CACHE_BASE = None
 def cache_dir():
-    base = os.environ.get("NOMADNET_IMAGE_CACHE")
+    global CONV_CACHE_BASE
-    if not base:
+    if not CONV_CACHE_BASE:
-        base = os.environ.get("XDG_CACHE_HOME") or os.path.join(os.path.expanduser("~"), ".cache")
+        from nomadnet.NomadNetworkApp import NomadNetworkApp
-        base = os.path.join(base, "nomadnet", "image-cache")
+        CONV_CACHE_BASE = NomadNetworkApp._shared_instance.dispcachepath
-    return base
+    return CONV_CACHE_BASE
 
 def _cache_path(source_data):
     return os.path.join(cache_dir(), hashlib.sha256(source_data).hexdigest() + ".png")

nomadnet/vendor/Scrollable.py

@@ -9,6 +9,7 @@
 # GNU General Public License for more details
 # http://www.gnu.org/licenses/gpl-3.0.txt
 
+import RNS
 import time
 import urwid
 from urwid.widget import BOX, FIXED, FLOW
@@ -371,6 +372,11 @@ class ScrollBar(urwid.WidgetDecoration):
         self._sevt_interval = 0
         self._sevt_start = 0
         self._sfreq_deque = deque(maxlen=5)
+        
+        self._debug_hz = 0
+        self._debug_hz_smoothed = 0
+        self._debug_scroll_ms = 0
+        self._debug_scroll_started = 0
 
     def render(self, size, focus=False):
         maxcol, maxrow = size
@@ -390,11 +396,24 @@ class ScrollBar(urwid.WidgetDecoration):
         ow_rows_max = ow_base.rows_max(ow_size, focus)
 
         ow_canv = ow.render(ow_size, focus)
+        
+        if self._debug_scroll_started:
+            total_ms = (time.time() - self._debug_scroll_started) * 1000
+            # RNS.log(f" accel={self._accel} , hz={RNS.prettyfrequency(self._debug_hz)} smoothed={RNS.prettyfrequency(self._debug_hz_smoothed)} scroll={self._debug_scroll_ms:.1f}ms total={total_ms:.1f}ms", RNS.LOG_DEBUG)
+            self._debug_scroll_started = 0
+        
         self._original_widget_size = ow_size
 
         pos = ow_base.get_scrollpos(ow_size, focus)
         posmax = ow_rows_max - maxrow
 
+        # Clamp position to valid scroll range. The position
+        # reported by the underlying widget can transiently
+        # fall outside [0, posmax] if its layout state is
+        # inconsistent.
+        if pos < 0: pos = 0
+        elif pos > posmax: pos = posmax
+
         # Thumb shrinks/grows according to the ratio of
         # <number of visible lines> / <number of total lines>
         thumb_weight = min(1, maxrow / max(1, ow_rows_max))
@@ -406,9 +425,12 @@ class ScrollBar(urwid.WidgetDecoration):
         if top_height == 0 and top_weight > 0:
             top_height = 1
 
-        # Bottom part is remaining space
+        # Bottom part is remaining space. Guard against
+        # negative values so the scrollbar canvas can
+        # never be taller than maxrow.
         bottom_height = maxrow - thumb_height - top_height
-        assert thumb_height + top_height + bottom_height == maxrow
+        if bottom_height < 0: bottom_height = 0
+        assert thumb_height + top_height + bottom_height <= maxrow
 
         # Create scrollbar canvas
         # Creating SolidCanvases of correct height may result in "cviews do not
@@ -427,10 +449,6 @@ class ScrollBar(urwid.WidgetDecoration):
         combinelist = [(ow_canv, None, True, ow_size[0]),
                        (sb_canv, None, False, sb_width)]
 
-
-
-
-                       
         self._sb_visible = True
         self._sb_thumb_height = thumb_height
         self._sb_ow_rows_max = ow_rows_max
@@ -531,23 +549,22 @@ class ScrollBar(urwid.WidgetDecoration):
                 sevt_freq = 1/self._sevt_interval
                 self._sfreq_deque.append(sevt_freq)
                 freq_smth = sum(self._sfreq_deque)/len(self._sfreq_deque)
+                self._debug_hz          = sevt_freq
+                self._debug_hz_smoothed = freq_smth
                 if len(self._sfreq_deque) >= self._sfreq_deque.maxlen:
                     if freq_smth > 25.0:  self._accel = max(self._accel, 3)
                     elif freq_smth > 12.0: self._accel = max(self._accel, 2)
                     else:                 self._accel = max(self._accel, 1)
                 
-                # import RNS; RNS.log(f"a={self._accel}  s={RNS.prettyfrequency(freq_smth)}  f={RNS.prettyfrequency(sevt_freq)}", RNS.LOG_CRITICAL)
-            
             step = self._accel
-            if button == 4:    # scroll wheel up
+            if button in (4, 5):
+                scroll_started = time.time()
                 pos = ow.get_scrollpos(ow_size)
-                newpos = pos - step
+                if button == 4: newpos = max(0, pos - step)
-                if newpos < 0: newpos = 0
+                else:           newpos = pos + step
                 ow.set_scrollpos(newpos)
-                return True
+                self._debug_scroll_started = scroll_started
-            elif button == 5:  # scroll wheel down
+                self._debug_scroll_ms      = (time.time() - scroll_started) * 1000
-                pos = ow.get_scrollpos(ow_size)
-                ow.set_scrollpos(pos + step)
                 return True
 
         return False

setup.py

@@ -36,6 +36,6 @@ setuptools.setup(
     entry_points= {
         'console_scripts': ['nomadnet=nomadnet.nomadnet:main']
     },
-    install_requires=["rns>=1.5.2", "lxmf>=1.1.1", "urwid>=3.0.5", "qrcode"],
+    install_requires=["rns>=1.5.5", "lxmf>=1.2.0", "urwid>=3.0.5", "qrcode"],
     python_requires=">=3.8",
 )