Updated logging

Fix msgpack import

MIRROR.md

@@ -0,0 +1,33 @@
+This repository is a public mirror. All potential future development is happening elsewhere.
+
+I am stepping back from all public-facing interaction with this project. Reticulum has always been primarily my work, and continuing in the current public, internet-facing model is no longer sustainable.
+
+The software remains available for use as-is. Occasional updates may appear at unpredictable intervals, but there will be no support, no responses to issues, no discussions, and no community management in this or any other public venue. If it doesn't work for you, it doesn't work. That is the entire extent of available troubleshooting assistance I can offer you.
+
+If you've followed this project for a while, you already know what this means. You know who designed, wrote and tested this, and you know how many years of my life it took. You'll also know about both my particular challenges and strengths, and how I believe anything worth building needs to be built and  maintained with our own hands.
+
+Seven months ago, I said I needed to step back, that I was exhausted, and that I needed to recover. I believed a public resolve would be enough to effectuate that, but while striving to get just a few more useful features and protocols out, the unproductive requests and demands also ramped up, and I got pulled back into the same patterns and draining interactions that I'd explicitly said I couldn't sustain anymore.
+
+So here's what you might have already guessed: I'm done playing the game by rules I can't win at.
+
+Everything you need is right here, and by any sensible measure, it's done. Anyone who wants to invest the time, skill and persistence can build on it, or completely re-imagine it with different priorities. That was always the point.
+
+The people who actually contributed - you know who you are, and you know I mean it when I say: Thank you. All of you who've used this to build something real - that was the goal, and you did it without needing me to hold your hand.
+
+The rest of you: You have what you need. Use it or don't. I am not going to be the person who explains it to you anymore.
+
+This is not a temporary break. It's not "see you after some rest", but a recognition that the current model is fundamentally incompatible with my life, my health, and my reality.
+
+If you want to support continued work, you can do so at the donation links listed in this repository. But please understand, that this is not purchasing support or guaranteeing updates. It is support for work that happens on my timeline, according to my capacity, which at the moment is not what it was.
+
+If you want Reticulum to continue evolving, you have the power to make that happen. The protocol is public domain. The code is open source. Everything you need is right here. I've provided the tools, but building what comes next is not my responsibility anymore. It's yours.
+
+To the small group of people who has actually been here, and understood what this work was and what it cost - you already know where to find me if it actually matters.
+
+To everyone else: This is where we part ways. No hard feelings. It's just time.
+
+---
+
+असतो मा सद्गमय
+तमसो मा ज्योतिर्गमय
+मृत्योर्मा अमृतं गमय

Makefile

@@ -21,5 +21,10 @@ build_wheel:
 release: remove_symlinks build_wheel create_symlinks
 
 upload:
+	@echo Ready to publish release over Reticulum
+	@read VOID
+	rngit release rns://7649a50d84610232d1416b41d2896aff/reticulum/nomadnet create $$(python setup.py --getversion):dist --name nomadnet
+
+upload-pip:
 	@echo Uploading to PyPi...
-	twine upload dist/*
+	twine upload dist/*.whl dist/*.tar.gz

README.md

@@ -1,8 +1,19 @@
 # Nomad Network - Communicate Freely
 
+> [!WARNING]  
+> Several hastily launched, incorrect, and in most cases entirely LLM-generated fakes of Nomad Network and associated tools are currently being circulated and marketed. Most of these "projects" violate the open source GPLv3 license that Nomad Network was published under, but claim independent ownership and license grants. Such claims or grants are **not** legally valid, and **not** recognized by the Nomad Network authors and copyright holders.
+>
+> Any claimed assertion of copyright or grant of license that such projects are making are [void grants](https://reticulum.network/manual/brandolinis.html#void-grants-legal-foundations-of-machine-generated-code). Both the violators themselves, and **all** downstream projects using the infringing derivatives, are **directly** and **personally** liable for the legal consequences.
+>
+> For more detailed information, see the relevant sections of the [Brandolini's Reference](https://reticulum.network/manual/brandolinis.html) chapter of the Reticulum Manual. For software and projects recognized by the Reticulum community, see the [Programs Using Reticulum](https://reticulum.network/manual/software.html) chapter.
+>
+> Do **not** use these "projects", they are a hazard to the entire ecosystem we have been carefully building over the last ten years.
+
+*This repository is [a public mirror](./MIRROR.md). All development is happening elsewhere.*
+
 Off-grid, resilient mesh communication with strong encryption, forward secrecy and extreme privacy.
 
-![Screenshot](https://github.com/markqvist/NomadNet/raw/master/docs/screenshots/1.png)
+![Screenshot](https://github.com/markqvist/NomadNet/raw/master/docs/screenshots/1a.png)
 
 Nomad Network allows you to build private and resilient communications platforms that are in complete control and ownership of the people that use them. No signups, no agreements, no handover of any data, no permissions and gatekeepers.
 
@@ -10,15 +21,23 @@ Nomad Network is build on [LXMF](https://github.com/markqvist/LXMF) and [Reticul
 
 Nomad Network does not need any connections to the public internet to work. In fact, it doesn't even need an IP or Ethernet network. You can use it entirely over packet radio, LoRa or even serial lines. But if you wish, you can bridge islanded networks over the Internet or private ethernet networks, or you can build networks running completely over the Internet. The choice is yours. Since Nomad Network uses Reticulum, it is efficient enough to run even over *extremely* low-bandwidth medium, and has been succesfully used over 300bps radio links.
 
-If you'd rather want to use an LXMF client with a graphical user interface, you may want to take a look at [Sideband](https://github.com/markqvist/sideband), which is available for Linux, Android and macOS.
+If you'd rather want to use an LXMF client with a graphical user interface, you may want to take a look at [Sideband](https://github.com/markqvist/sideband), which is available for Linux, Android, Windows and macOS.
+
+Other Nomad Network clients include:
+
+- [MeshChatX](https://meshchatx.com/)
+- [Columba (mobile)](https://github.com/torlando-tech/columba)
+- [Ren Browser](https://github.com/Quad4-Software/Ren-Browser)
 
 ## Notable Features
  - Encrypted messaging over packet-radio, LoRa, WiFi or anything else [Reticulum](https://github.com/markqvist/Reticulum) supports.
  - Zero-configuration, minimal-infrastructure mesh communication
  - Distributed and encrypted message store holds messages for offline users
  - Connectable nodes that can host pages and files
+ - Image rendering support in terminals that support it
  - Node-side generated pages with PHP, Python, bash or others
  - Built-in text-based browser for interacting with contents on nodes
+ - Built-in RRC client for live, many-to-many chat
  - An easy to use and bandwidth efficient markup language for writing pages
  - Page caching in browser
 
@@ -74,10 +93,10 @@ The first time the program is running, you will be presented with the **Guide se
 
 To use Nomad Network on packet radio or LoRa, you will need to configure your Reticulum installation to use any relevant packet radio TNCs or LoRa devices on your system. See the [Reticulum documentation](https://markqvist.github.io/Reticulum/manual/interfaces.html) for info. For a general introduction on how to set up such a system, take a look at [this post](https://unsigned.io/private-messaging-over-lora/).
 
-If you want to try Nomad Network without building your own physical network, you can connect to the [Unsigned.io RNS Testnet](https://github.com/markqvist/Reticulum#public-testnet) over the Internet, where there is already some Nomad Network and LXMF activity. If you connect to the testnet, you can leave nomadnet running for a while and wait for it to receive announces from other nodes on the network that host pages or services, or you can try connecting directly to some nodes listed here:
+If you want to try Nomad Network without building your own physical network, you can connect to the [distributed backbone](https://reticulum.network/manual/gettingstartedfast.html#bootstrapping-connectivity) over the Internet, where there is already quite a bit of Nomad Network and LXMF activity. If you connect to the testnet, you can leave nomadnet running for a while and wait for it to receive announces from other nodes on the network that host pages or services, or you can try connecting directly to some nodes listed here:
 
- - `abb3ebcd03cb2388a838e70c001291f9` Dublin Hub Testnet Node
+ - `9ce92808be498e9e05590ff27cbfdfe4` The rns.recipes forum
- - `ea6a715f814bdc37e56f80c34da6ad51` Frankfurt Hub Testnet Node
+ - `a4a5e861626ce97c9aa544d9ecdf6d22` rmap.world
 
 To browse pages on a node that is not currently known, open the URL dialog in the `Network` section of the program by pressing `Ctrl+U`, paste or enter the address and select `Go` or press enter. Nomadnet will attempt to discover and connect to the requested node.
 
@@ -86,42 +105,12 @@ You can install Nomad Network on Android using Termux, but there's a few more co
 
 For a native Android application with a graphical user interface, have a look at [Sideband](https://github.com/markqvist/Sideband).
 
-### Docker Images
-
-Nomad Network is automatically published as a docker image on Github Packages. Image tags are one of either `master` (for the very latest commit) or the version number (eg `0.2.0`) for a specific release.
-
-```sh
-$ docker pull ghcr.io/markqvist/nomadnet:master
-
-# Run nomadnet interactively in a container
-$ docker run -it ghcr.io/markqvist/nomadnet:master --textui
-
-# Run nomadnet as a daemon, using config stored on the host machine in specified
-# directories, and connect the containers network to the host network (which will
-# allow the default AutoInterface to automatically peer with other discovered
-# Reticulum instances).
-$ docker run -d \
-  -v /local/path/nomadnetconfigdir/:/root/.nomadnetwork/ \
-  -v /local/path/reticulumconfigdir/:/root/.reticulum/ \
-  --network host
-  ghcr.io/markqvist/nomadnet:master
-
-# You can also keep the network of the container isolated from the host, but you
-# will need to manually configure one or more Reticulum interfaces to reach other
-# nodes in a network, by editing the Reticulum configuration file.
-$ docker run -d \
-  -v /local/path/nomadnetconfigdir/:/root/.nomadnetwork/ \
-  -v /local/path/reticulumconfigdir/:/root/.reticulum/ \
-  ghcr.io/markqvist/nomadnet:master
-
-# Send daemon log output to console instead of file
-$ docker run -i ghcr.io/markqvist/nomadnet:master --daemon --console
-```
 
 ## Tools & Extensions
 
 Nomad Network is a very flexible and extensible platform, and a variety of community-provided tools, utilities and node-side extensions exist:
 
+- [Retipedia](https://github.com/RFnexus/Retipedia)
 - [NomadForum](https://codeberg.org/AutumnSpark1226/nomadForum) ([GitHub mirror](https://github.com/AutumnSpark1226/nomadForum))
 - [NomadForecast](https://github.com/faragher/NomadForecast)
 - [micron-blog](https://github.com/randogoth/micron-blog)
@@ -135,9 +124,6 @@ Nomad Network is a very flexible and extensible platform, and a variety of commu
 - [POPR](https://github.com/faragher/POPR)
 - [LXMF Tools](https://github.com/SebastianObi/LXMF-Tools)
 
-## Help & Discussion
-
-For help requests, discussion, sharing ideas or anything else related to Nomad Network, please have a look at the [Nomad Network discussions pages](https://github.com/markqvist/Reticulum/discussions/categories/nomad-network).
 
 ## Support Nomad Network
 You can help support the continued development of open, free and private communications systems by donating via one of the following channels:
@@ -148,42 +134,23 @@ You can help support the continued development of open, free and private communi
   ```
 - Bitcoin
   ```
-  bc1p4a6axuvl7n9hpapfj8sv5reqj8kz6uxa67d5en70vzrttj0fmcusgxsfk5
+  bc1pgqgu8h8xvj4jtafslq396v7ju7hkgymyrzyqft4llfslz5vp99psqfk3a6
   ```
 - Ethereum
   ```
-  0xae89F3B94fC4AD6563F0864a55F9a697a90261ff
+  0x91C421DdfB8a30a49A71d63447ddb54cEBe3465E
   ```
+- Liberapay: https://liberapay.com/Reticulum/
+
 - Ko-Fi: https://ko-fi.com/markqvist
 
-## Development Roadmap
-
-- New major features
-    - Network-wide propagated bulletins and discussion threads
-    - Collaborative maps and geospatial information sharing
-- Minor improvements and fixes
-    - Link status (RSSI and SNR) in conversation or conv list
-    - Ctrl-M shorcut for jumping to menu
-    - Share node with other users / send node info to user
-    - Fix internal editor failing on some OSes with no "editor" alias
-    - Possibly add a required-width header
-    - Improve browser handling of remote link close
-    - Better navigation handling when requests fail (also because of closed links)
-    - Retry failed messages mechanism
-    - Re-arrange buttons to be more consistent
-    - Input field for pages
-    - Post mechanism
-    - Term compatibility notice in readme
-    - Selected icon in conversation list
-    - Possibly a Search Local Nodes function
-    - Possibly add via entry in node info box, next to distance
 
 ## Caveat Emptor
-Nomad Network is beta software, and should be considered as such. While it has been built with cryptography best-practices very foremost in mind, it _has not_ been externally security audited, and there could very well be privacy-breaking bugs. If you want to help out, or help sponsor an audit, please do get in touch.
+Nomad Network is beta software, and should be considered as such. Use at your own risk and responsibility.
 
 ## Screenshots
 
-![Screenshot 1](https://github.com/markqvist/NomadNet/raw/master/docs/screenshots/1.png)
+![Screenshot 1](https://github.com/markqvist/NomadNet/raw/master/docs/screenshots/1a.png)
 
 ![Screenshot 2](https://github.com/markqvist/NomadNet/raw/master/docs/screenshots/2.png)
 
@@ -192,3 +159,5 @@ Nomad Network is beta software, and should be considered as such. While it has b
 ![Screenshot 4](https://github.com/markqvist/NomadNet/raw/master/docs/screenshots/4.png)
 
 ![Screenshot 5](https://github.com/markqvist/NomadNet/raw/master/docs/screenshots/5.png)
+
+![Screenshot 6](https://github.com/markqvist/NomadNet/raw/master/docs/screenshots/6.png)

README.mu

@@ -0,0 +1,123 @@
+>> Nomad Network - Communicate Freely
+
+Off-grid, resilient mesh communication with strong encryption, forward secrecy and extreme privacy.
+
+Nomad Network allows you to build private and resilient communications platforms that are in complete control and ownership of the people that use them. No signups, no agreements, no handover of any data, no permissions and gatekeepers.
+
+Nomad Network is build on `_`!`[LXMF`a8d24177d946de4f1f0a0fe1af9a1338:/page/repo.mu`g=reticulum|r=lxmf]`!`_ and `_`!`[Reticulum`a8d24177d946de4f1f0a0fe1af9a1338:/page/repo.mu`g=reticulum|r=reticulum]`!`_, which together provides the cryptographic mesh functionality and peer-to-peer message routing that Nomad Network relies on. This foundation also makes it possible to use the program over a very wide variety of communication mediums, from packet radio to fiber optics.
+
+Nomad Network does not need any connections to the public internet to work. In fact, it doesn't even need an IP or Ethernet network. You can use it entirely over packet radio, LoRa or even serial lines. But if you wish, you can bridge islanded networks over the Internet or private ethernet networks, or you can build networks running completely over the Internet. The choice is yours. Since Nomad Network uses Reticulum, it is efficient enough to run even over `*extremely`* low-bandwidth medium, and has been succesfully used over 300bps radio links.
+
+If you'd rather want to use an LXMF client with a graphical user interface, you may want to take a look at `_`!`[Sideband`a8d24177d946de4f1f0a0fe1af9a1338:/page/repo.mu`g=reticulum|r=sideband]`!`_, which is available for Linux, Android, Windows and macOS.
+
+>> Notable Features
+
+ • Encrypted messaging over packet-radio, LoRa, WiFi or anything else `_`!`[Reticulum`a8d24177d946de4f1f0a0fe1af9a1338:/page/repo.mu`g=reticulum|r=reticulum]`!`_ supports.
+ • Zero-configuration, minimal-infrastructure mesh communication
+ • Distributed and encrypted message store holds messages for offline users
+ • Connectable nodes that can host pages and files
+ • Image rendering support in terminals that support it
+ • Node-side generated pages with PHP, Python, bash or others
+ • Built-in text-based browser for interacting with contents on nodes
+ • Built-in RRC client for live, many-to-many chat
+ • An easy to use and bandwidth efficient markup language for writing pages
+ • Page caching in browser
+
+>> How do I get started?
+
+The easiest way to install Nomad Network is via `B333pip`b:
+
+`B333
+`=
+# Install Nomad Network and dependencies
+pip install nomadnet
+
+# Run the client
+nomadnet
+
+# Or alternatively run as a daemon, with no user interface
+nomadnet --daemon
+
+# List options
+nomadnet --help
+`=
+`b
+
+If you are using an operating system that blocks normal user package installation via `B333pip`b, you can return `B333pip`b to normal behaviour by editing the `B333~/.config/pip/pip.conf`b file, and adding the following directive in the `B333[global]`b section:
+
+`B333
+`=
+[global]
+break-system-packages = true
+`=
+`b
+
+Alternatively, you can use the `B333pipx`b tool to install Nomad Network in an isolated environment:
+
+`B333
+`=
+# Install Nomad Network
+pipx install nomadnet
+
+# Optionally install Reticulum utilities
+pipx install rns
+
+# Optionally install standalone LXMF utilities
+pipx install lxmf
+
+# Run the client
+nomadnet
+
+# Or alternatively run as a daemon, with no user interface
+nomadnet --daemon
+
+# List options
+nomadnet --help
+`=
+`b
+
+`!Please Note`!: If this is the very first time you use pip to install a program on your system, you might need to reboot your system for the program to become available. If you get a "command not found" error or similar when running the program, reboot your system and try again.
+
+The first time the program is running, you will be presented with the `!Guide section`!, which contains all the information you need to start using Nomad Network.
+
+To use Nomad Network on packet radio or LoRa, you will need to configure your Reticulum installation to use any relevant packet radio TNCs or LoRa devices on your system. See the `_`!`[Reticulum documentation`a8d24177d946de4f1f0a0fe1af9a1338:/page/blob.mu`g=reticulum|r=reticulum|ref=HEAD|path=docs/markdown/index.md]`!`_ for info.
+
+If you want to try Nomad Network without building your own physical network, you can connect to the `_`!`[distributed RNS backbone`a8d24177d946de4f1f0a0fe1af9a1338:/page/blob.mu`g=reticulum|r=reticulum|ref=HEAD|path=docs/markdown/gettingstartedfast.md|anchor=connect-to-the-distributed-backbone]`!`_ over the Internet, where there is already quite a bit of Nomad Network and LXMF activity. If you connect to the testnet, you can leave nomadnet running for a while and wait for it to receive announces from other nodes on the network that host pages or services, or you can try connecting directly to some nodes listed here:
+
+ • `!`_`[9ce92808be498e9e05590ff27cbfdfe4]`_`! The rns.recipes forum
+ • `!`_`[a4a5e861626ce97c9aa544d9ecdf6d22]`_`! rmap.world
+
+To browse pages on a node that is not currently known, open the URL dialog in the `!Network`! section of the program by pressing `!Ctrl+U`!, paste or enter the address and select `!Go`! or press enter. Nomadnet will attempt to discover and connect to the requested node.
+
+>>> Install on Android
+
+You can install Nomad Network on Android using Termux, but there's a few more commands involved than the above one-liner. The process is documented in the `_`!`[Android Installation`a8d24177d946de4f1f0a0fe1af9a1338:/page/blob.mu`g=reticulum|r=reticulum|ref=HEAD|path=docs/markdown/gettingstartedfast.md|anchor=reticulum-on-android]`!`_ section of the Reticulum Manual. Once the Reticulum has been installed according to the linked documentation, Nomad Network can be installed as usual with pip.
+
+For a native Android application with a graphical user interface, have a look at `_`!`[Sideband`a8d24177d946de4f1f0a0fe1af9a1338:/page/repo.mu`g=reticulum|r=sideband]`!`_.
+
+>> Help & Discussion
+
+For help requests, discussion, sharing ideas or anything else related to Nomad Network, please have a look at the `_`!`[rns.recipes forum`9ce92808be498e9e05590ff27cbfdfe4]`!`_.
+
+>> Support Nomad Network
+
+For this to be possible, I need your help. Please support the continued development of open, free and private communications systems by donating via one of the following channels:
+
+• `!Monero`!
+  84FpY1QbxHcgdseePYNmhTHcrgMX4nFfBYtz2GKYToqHVVhJp8Eaw1Z1EedRnKD19b3B8NiLCGVxzKV17UMmmeEsCrPyA5w
+
+• `!Bitcoin`!
+  bc1pgqgu8h8xvj4jtafslq396v7ju7hkgymyrzyqft4llfslz5vp99psqfk3a6
+
+• `!Ethereum`!
+  0x91C421DdfB8a30a49A71d63447ddb54cEBe3465E
+
+• `!Liberapay`!
+  `[https://liberapay.com/Reticulum/]
+
+• `!Ko-Fi`!
+  `[https://ko-fi.com/markqvist]
+
+>> Caveat Emptor
+
+Nomad Network is experimental software, and should be considered as such. Use at your own risk and responsibility.

nomadnet/Conversation.py

@@ -1,42 +1,48 @@
 import os
+import re
 import RNS
 import LXMF
 import shutil
+import RNS.vendor.umsgpack as msgpack
 import nomadnet
 from nomadnet.Directory import DirectoryEntry
+from LXMF import display_name_from_app_data
 
 class Conversation:
     cached_conversations = {}
     unread_conversations = {}
+    failed_conversations = {}
     created_callback = None
 
     aspect_filter = "lxmf.delivery"
+    receive_path_responses = True
     @staticmethod
-    def received_announce(destination_hash, announced_identity, app_data):
+    def received_announce(destination_hash, announced_identity, app_data, announce_packet_hash=None, is_path_response=False):
         app = nomadnet.NomadNetworkApp.get_shared_instance()
 
+        if not display_name_from_app_data(app_data):
+            RNS.log("Ignored invalid lxmf.delivery announce from "+RNS.prettyhexrep(destination_hash), RNS.LOG_DEBUG)
+
         if not destination_hash in app.ignored_list:
             destination_hash_text = RNS.hexrep(destination_hash, delimit=False)
-            # Check if the announced destination is in
+            # Notify if the announced destination is one of our
-            # our list of conversations
+            # existing conversations. Checking the directory directly
-            if destination_hash_text in [e[0] for e in Conversation.conversation_list(app)]:
+            # avoids scanning and opening every conversation on disk
-                if app.directory.find(destination_hash):
+            # for each announce received.
-                    if Conversation.created_callback != None:
+            if os.path.isdir(app.conversationpath + "/" + destination_hash_text):
-                        Conversation.created_callback()
+                if Conversation.created_callback != None:
-                else:
+                    Conversation.created_callback()
-                    if Conversation.created_callback != None:
-                        Conversation.created_callback()
 
-            # This reformats the new v0.5.0 announce data back to the expected format
+            if not is_path_response:
-            # for nomadnets storage and other handling functions.
+                # This reformats the new v0.5.0 announce data back to the expected format
-            dn = LXMF.display_name_from_app_data(app_data)
+                # for nomadnets storage and other handling functions.
-            app_data = b""
+                dn = LXMF.display_name_from_app_data(app_data)
-            if dn != None:
+                app_data = b""
-                app_data = dn.encode("utf-8")
+                if dn != None: app_data = dn.encode("utf-8")
 
-            # Add the announce to the directory announce
+                # Add the announce to the directory announce
-            # stream logger
+                # stream logger
-            app.directory.lxmf_announce_received(destination_hash, app_data)
+                app.directory.lxmf_announce_received(destination_hash, app_data)
 
         else:
             RNS.log("Ignored announce from "+RNS.prettyhexrep(destination_hash), RNS.LOG_DEBUG)
@@ -66,20 +72,64 @@ class Conversation:
 
         ingested_path = lxmessage.write_to_directory(conversation_path)
 
+        if not originator and ingested_path is not None:
+            try:
+                rssi = getattr(lxmessage, "rssi", None)
+                snr = getattr(lxmessage, "snr", None)
+                q = getattr(lxmessage, "q", None)
+                if rssi is not None or snr is not None or q is not None:
+                    index = ConversationMessage.read_index(conversation_path)
+                    filename = os.path.basename(ingested_path)
+                    entry = index.get(filename) or {}
+                    if rssi is not None: entry["rssi"] = rssi
+                    if snr is not None: entry["snr"] = snr
+                    if q is not None: entry["q"] = q
+                    index[filename] = entry
+                    with open(os.path.join(conversation_path, ".index"), "wb") as index_file:
+                        index_file.write(msgpack.packb(index))
+            except Exception as e:
+                RNS.log("Error while storing reception stats for message: "+str(e), RNS.LOG_ERROR)
+
+        try:
+            ConversationMessage.extract_attachments_from_lxm(lxmessage, app)
+        except Exception as e:
+            RNS.log("Error extracting attachments: "+str(e), RNS.LOG_ERROR)
+
         if RNS.hexrep(source_hash, delimit=False) in Conversation.cached_conversations:
             conversation = Conversation.cached_conversations[RNS.hexrep(source_hash, delimit=False)]
             conversation.scan_storage()
 
-        if not source_hash in Conversation.unread_conversations:
+        lxm_state = getattr(lxmessage, "state", None)
-            Conversation.unread_conversations[source_hash] = True
+        is_failed = lxm_state in (LXMF.LXMessage.FAILED, LXMF.LXMessage.REJECTED)
+
+        if not originator:
+            if source_hash in Conversation.unread_conversations:
+                Conversation.unread_conversations[source_hash] += 1
+            else:
+                Conversation.unread_conversations[source_hash] = 1
+
             try:
                 dirname = RNS.hexrep(source_hash, delimit=False)
-                open(app.conversationpath + "/" + dirname + "/unread", 'a').close()
+                with open(app.conversationpath + "/" + dirname + "/unread", "w") as uf:
-            except Exception as e:
+                    uf.write(str(Conversation.unread_conversations[source_hash]))
+            except Exception:
                 pass
 
-            if Conversation.created_callback != None:
+        elif is_failed:
-                Conversation.created_callback()
+            if source_hash in Conversation.failed_conversations:
+                Conversation.failed_conversations[source_hash] += 1
+            else:
+                Conversation.failed_conversations[source_hash] = 1
+
+            try:
+                dirname = RNS.hexrep(source_hash, delimit=False)
+                with open(app.conversationpath + "/" + dirname + "/failed", "w") as ff:
+                    ff.write(str(Conversation.failed_conversations[source_hash]))
+            except Exception:
+                pass
+
+        if Conversation.created_callback != None:
+            Conversation.created_callback()
 
         return ingested_path
 
@@ -94,12 +144,29 @@ class Conversation:
                     app_data         = RNS.Identity.recall_app_data(source_hash)
                     display_name     = app.directory.display_name(source_hash)
 
-                    unread = False
+                    unread = 0
                     if source_hash in Conversation.unread_conversations:
-                        unread = True
+                        unread = Conversation.unread_conversations[source_hash]
                     elif os.path.isfile(app.conversationpath + "/" + dirname + "/unread"):
-                        Conversation.unread_conversations[source_hash] = True
+                        try:
-                        unread = True
+                            with open(app.conversationpath + "/" + dirname + "/unread", "r") as uf:
+                                content = uf.read().strip()
+                                unread = int(content) if content else 1
+                        except Exception:
+                            unread = 1
+                        Conversation.unread_conversations[source_hash] = unread
+
+                    failed = 0
+                    if source_hash in Conversation.failed_conversations:
+                        failed = Conversation.failed_conversations[source_hash]
+                    elif os.path.isfile(app.conversationpath + "/" + dirname + "/failed"):
+                        try:
+                            with open(app.conversationpath + "/" + dirname + "/failed", "r") as ff:
+                                content = ff.read().strip()
+                                failed = int(content) if content else 1
+                        except Exception:
+                            failed = 1
+                        Conversation.failed_conversations[source_hash] = failed
 
                     if display_name == None and app_data:
                         display_name = LXMF.display_name_from_app_data(app_data)
@@ -111,13 +178,19 @@ class Conversation:
 
                     trust_level      = app.directory.trust_level(source_hash, display_name)
 
-                    entry = (source_hash_text, display_name, trust_level, sort_name, unread)
+                    conversation_dir = app.conversationpath + "/" + dirname
+                    try:
+                        last_activity = os.path.getmtime(conversation_dir)
+                    except Exception:
+                        last_activity = 0
+
+                    entry = (source_hash_text, display_name, trust_level, sort_name, unread, last_activity, failed)
                     conversations.append(entry)
 
                 except Exception as e:
                     RNS.log("Error while loading conversation "+str(dirname)+", skipping it. The contained exception was: "+str(e), RNS.LOG_ERROR)
 
-        conversations.sort(key=lambda e: (-e[2], e[3], e[0]), reverse=False)
+        conversations.sort(key=lambda e: e[5], reverse=True)
 
         return conversations
 
@@ -149,15 +222,9 @@ class Conversation:
 
         self.__changed_callback = None
 
-        if not RNS.Identity.recall(bytes.fromhex(self.source_hash)):
+        if not self.ensure_send_destination():
             RNS.Transport.request_path(bytes.fromhex(source_hash))
 
-        self.source_identity = RNS.Identity.recall(bytes.fromhex(self.source_hash))
-
-        if self.source_identity:
-            self.source_known = True
-            self.send_destination = RNS.Destination(self.source_identity, RNS.Destination.OUT, RNS.Destination.SINGLE, "lxmf", "delivery")
-
         if initiator:
             if not os.path.isdir(self.messages_path):
                 os.makedirs(self.messages_path)
@@ -172,14 +239,46 @@ class Conversation:
 
     def scan_storage(self):
         old_len = len(self.messages)
+        existing = {}
+        for msg in self.messages:
+            existing[msg.file_path] = msg
+
+        index = ConversationMessage.read_index(self.messages_path)
+
         self.messages = []
         for filename in os.listdir(self.messages_path):
             if len(filename) == RNS.Identity.HASHLENGTH//8*2:
                 message_path = self.messages_path + "/" + filename
-                self.messages.append(ConversationMessage(message_path))
+                if message_path in existing:
+                    old_msg = existing[message_path]
+                    try:
+                        current_mtime = os.path.getmtime(message_path)
+                        if current_mtime > old_msg.sort_timestamp:
+                            old_msg._cached_state = None
+                            old_msg._cached_method = None
+                            old_msg.sort_timestamp = current_mtime
+                    except Exception:
+                        pass
+                    self.messages.append(old_msg)
+                else:
+                    msg = ConversationMessage(message_path)
+                    if filename in index:
+                        msg.restore_from_index(index[filename])
+                    self.messages.append(msg)
 
         new_len = len(self.messages)
 
+        needs_index_update = []
+        for msg in self.messages:
+            filename = os.path.basename(msg.file_path)
+            if msg._cached_state is not None:
+                if filename not in index:
+                    needs_index_update.append(msg)
+                elif "content" not in index[filename]:
+                    needs_index_update.append(msg)
+        if needs_index_update:
+            ConversationMessage.write_index(self.messages_path, needs_index_update)
+
         if new_len > old_len:
             self.unread = True
 
@@ -208,14 +307,27 @@ class Conversation:
     def register_changed_callback(self, callback):
         self.__changed_callback = callback
 
-    def send(self, content="", title=""):
+    def ensure_send_destination(self):
-        if self.send_destination:
+        if self.send_destination == None:
+            self.source_identity = RNS.Identity.recall(bytes.fromhex(self.source_hash))
+
+            if self.source_identity:
+                self.source_known = True
+                self.send_destination = RNS.Destination(self.source_identity, RNS.Destination.OUT, RNS.Destination.SINGLE, "lxmf", "delivery")
+
+        return self.send_destination != None
+
+    def send(self, content="", title="", fields=None):
+        if self.ensure_send_destination():
             dest = self.send_destination
             source = self.app.lxmf_destination
             desired_method = LXMF.LXMessage.DIRECT
-            if self.app.directory.preferred_delivery(dest.hash) == DirectoryEntry.PROPAGATED:
+            preferred = self.app.directory.preferred_delivery(dest.hash)
+            if preferred == DirectoryEntry.PROPAGATED:
                 if self.app.message_router.get_outbound_propagation_node() != None:
                     desired_method = LXMF.LXMessage.PROPAGATED
+            elif preferred == DirectoryEntry.OPPORTUNISTIC:
+                desired_method = LXMF.LXMessage.OPPORTUNISTIC
             else:
                 if not self.app.message_router.delivery_link_available(dest.hash) and RNS.Identity.current_ratchet_id(dest.hash) != None:
                     RNS.log(f"Have ratchet for {RNS.prettyhexrep(dest.hash)}, requesting opportunistic delivery of message", RNS.LOG_DEBUG)
@@ -225,7 +337,7 @@ class Conversation:
             if self.app.directory.trust_level(dest.hash) == DirectoryEntry.TRUSTED:
                 dest_is_trusted = True
 
-            lxm = LXMF.LXMessage(dest, source, content, title=title, desired_method=desired_method, include_ticket=dest_is_trusted)
+            lxm = LXMF.LXMessage(dest, source, content, title=title, fields=fields, desired_method=desired_method, include_ticket=dest_is_trusted)
             lxm.register_delivery_callback(self.message_notification)
             lxm.register_failed_callback(self.message_notification)
 
@@ -239,11 +351,12 @@ class Conversation:
 
             return True
         else:
-            RNS.log("Destination is not known, cannot create LXMF Message.", RNS.LOG_VERBOSE)
+            RNS.log("Destination is not known, cannot create LXMF Message. Requesting path...", RNS.LOG_VERBOSE)
+            RNS.Transport.request_path(bytes.fromhex(self.source_hash))
             return False
 
     def paper_output(self, content="", title="", mode="print_qr"):
-        if self.send_destination:
+        if self.ensure_send_destination():
             try:
                 dest = self.send_destination
                 source = self.app.lxmf_destination
@@ -324,19 +437,52 @@ class Conversation:
 
 
 class ConversationMessage:
+    # Sentinel marking the renderer as not yet cached, so that a real
+    # cached value of None (no renderer field) can be told apart from a
+    # cache miss without forcing a disk load.
+    _RENDERER_UNSET = object()
+
     def __init__(self, file_path):
         self.file_path = file_path
         self.loaded    = False
         self.timestamp = None
         self.lxm       = None
 
+        self._cached_renderer = ConversationMessage._RENDERER_UNSET
+        self._cached_hash = None
+        self._cached_state = None
+        self._cached_title = None
+        self._cached_content = None
+        self._cached_source_hash = None
+        self._cached_transport_encrypted = None
+        self._cached_transport_encryption = None
+        self._cached_signature_validated = None
+        self._cached_unverified_reason = None
+        self._cached_method = None
+        self._cached_has_attachments = None
+        self._cached_attachment_names = None
+        self._cached_rssi = None
+        self._cached_snr = None
+        self._cached_q = None
+
+        self.sort_timestamp = os.path.getmtime(file_path) if os.path.isfile(file_path) else 0
+
+        filename = os.path.basename(file_path)
+        if len(filename) == RNS.Identity.HASHLENGTH//8*2:
+            try:
+                self._cached_hash = bytes.fromhex(filename)
+            except Exception:
+                pass
+
     def load(self):
         try:
-            self.lxm = LXMF.LXMessage.unpack_from_file(open(self.file_path, "rb"))
+            with open(self.file_path, "rb") as lxm_file:
+                self.lxm = LXMF.LXMessage.unpack_from_file(lxm_file)
             self.loaded = True
             self.timestamp = self.lxm.timestamp
             self.sort_timestamp = os.path.getmtime(self.file_path)
 
+
             if self.lxm.state > LXMF.LXMessage.GENERATING and self.lxm.state < LXMF.LXMessage.SENT:
                 found = False
 
@@ -351,6 +497,95 @@ class ConversationMessage:
                 if not found:
                     self.lxm.state = LXMF.LXMessage.FAILED
 
+            if self._cached_hash is None:
+                self._cached_hash = self.lxm.hash
+            self._cached_state = self.lxm.state
+            if self._cached_source_hash is None:
+                self._cached_source_hash = self.lxm.source_hash
+            self._cached_transport_encrypted = self.lxm.transport_encrypted
+            self._cached_transport_encryption = self.lxm.transport_encryption
+            if self._cached_signature_validated is None:
+                self._cached_signature_validated = self.lxm.signature_validated
+            self._cached_method = self.lxm.method
+            if self._cached_rssi is None:
+                self._cached_rssi = getattr(self.lxm, "rssi", None)
+            if self._cached_snr is None:
+                self._cached_snr = getattr(self.lxm, "snr", None)
+            if self._cached_q is None:
+                self._cached_q = getattr(self.lxm, "q", None)
+            if hasattr(self.lxm, "get_fields"):
+                _fields = self.lxm.get_fields()
+                if _fields and isinstance(_fields, dict) and LXMF.FIELD_RENDERER in _fields:
+                    self._cached_renderer = _fields[LXMF.FIELD_RENDERER]
+                else:
+                    self._cached_renderer = None
+            else:
+                self._cached_renderer = None
+            if self._cached_unverified_reason is None and hasattr(self.lxm, "unverified_reason"):
+                self._cached_unverified_reason = self.lxm.unverified_reason
+            self._cached_title = self.lxm.title_as_string()
+            self._cached_content = self.lxm.content_as_string()
+
+            if self._cached_has_attachments is None:
+                found_in_fields = False
+                fields = None
+                if hasattr(self.lxm, "get_fields"):
+                    fields = self.lxm.get_fields()
+                if fields and isinstance(fields, dict):
+                    found_in_fields = (
+                        LXMF.FIELD_FILE_ATTACHMENTS in fields
+                        or LXMF.FIELD_IMAGE in fields
+                        or LXMF.FIELD_AUDIO in fields
+                    )
+                    if found_in_fields:
+                        names = []
+                        file_atts = fields.get(LXMF.FIELD_FILE_ATTACHMENTS, [])
+                        for idx, att in enumerate(file_atts):
+                            if isinstance(att, list) and len(att) >= 2:
+                                size = len(att[1]) if isinstance(att[1], bytes) else 0
+                                safe = ConversationMessage.safe_attachment_name(att[0], fallback="attachment_"+str(idx))
+                                names.append(("file", safe, size))
+                        if LXMF.FIELD_IMAGE in fields:
+                            fmt, data = ConversationMessage._unpack_media_field(fields[LXMF.FIELD_IMAGE])
+                            if data:
+                                size = len(data)
+                                ext = ConversationMessage._ext_from_media_format(fmt, data)
+                                safe = ConversationMessage.safe_attachment_name("image"+ext, fallback="image")
+                                names.append(("file", safe, size))
+                        if LXMF.FIELD_AUDIO in fields:
+                            fmt, data = ConversationMessage._unpack_media_field(fields[LXMF.FIELD_AUDIO])
+                            if data:
+                                size = len(data)
+                                ext = ConversationMessage._ext_from_media_format(fmt, data, is_audio=True)
+                                safe = ConversationMessage.safe_attachment_name("audio"+ext, fallback="audio")
+                                names.append(("file", safe, size))
+                        self._cached_has_attachments = True
+                        self._cached_attachment_names = names
+
+                if not found_in_fields:
+                    att_dir = self._attachment_dir()
+                    if att_dir and os.path.isdir(att_dir):
+                        manifest = self._read_attachment_manifest(att_dir)
+                        if manifest and "files" in manifest and len(manifest["files"]) > 0:
+                            names = []
+                            for entry in manifest["files"]:
+                                names.append(("file", entry["name"], entry.get("size", 0)))
+                            self._cached_has_attachments = True
+                            self._cached_attachment_names = names
+                        else:
+                            self._cached_has_attachments = False
+                            self._cached_attachment_names = []
+                    else:
+                        self._cached_has_attachments = False
+                        self._cached_attachment_names = []
+
+            try:
+                app = nomadnet.NomadNetworkApp.get_shared_instance()
+                ConversationMessage.extract_attachments_from_lxm(self.lxm, app)
+                ConversationMessage.strip_attachments_from_file(self.file_path, app)
+            except Exception:
+                pass
+
         except Exception as e:
             RNS.log("Error while loading LXMF message "+str(self.file_path)+" from disk. The contained exception was: "+str(e), RNS.LOG_ERROR)
 
@@ -364,60 +599,470 @@ class ConversationMessage:
             os.unlink(self.file_path)
 
     def get_timestamp(self):
+        if self.timestamp is not None:
+            return self.timestamp
         if not self.loaded:
             self.load()
-
         return self.timestamp
 
     def get_title(self):
+        if self._cached_title is not None:
+            return self._cached_title
         if not self.loaded:
             self.load()
-
+        return self._cached_title if self._cached_title is not None else ""
-        return self.lxm.title_as_string()
 
     def get_content(self):
+        if self._cached_content is not None:
+            return self._cached_content
         if not self.loaded:
             self.load()
-
+        return self._cached_content if self._cached_content is not None else ""
-        return self.lxm.content_as_string()
 
     def get_hash(self):
+        if self._cached_hash is not None:
+            return self._cached_hash
         if not self.loaded:
             self.load()
-
+        return self._cached_hash
-        return self.lxm.hash
 
     def get_state(self):
+        if self._cached_state is not None:
+            return self._cached_state
         if not self.loaded:
             self.load()
+        return self._cached_state
 
-        return self.lxm.state
+    def get_method(self):
+        if self._cached_method is not None:
+            return self._cached_method
+        if not self.loaded:
+            self.load()
+        return self._cached_method
+
+    def get_rssi(self):
+        return self._cached_rssi
+
+    def get_snr(self):
+        return self._cached_snr
+
+    def get_q(self):
+        return self._cached_q
 
     def get_transport_encryption(self):
+        if self._cached_transport_encryption is not None:
+            return self._cached_transport_encryption
         if not self.loaded:
             self.load()
-
+        return self._cached_transport_encryption
-        return self.lxm.transport_encryption
 
     def get_transport_encrypted(self):
+        if self._cached_transport_encrypted is not None:
+            return self._cached_transport_encrypted
         if not self.loaded:
             self.load()
-
+        return self._cached_transport_encrypted
-        return self.lxm.transport_encrypted
 
     def signature_validated(self):
+        if self._cached_signature_validated is not None:
+            return self._cached_signature_validated
         if not self.loaded:
             self.load()
-
+        return self._cached_signature_validated
-        return self.lxm.signature_validated
 
     def get_signature_description(self):
         if self.signature_validated():
             return "Signature Verified"
         else:
-            if self.lxm.unverified_reason == LXMF.LXMessage.SOURCE_UNKNOWN:
+            reason = self._cached_unverified_reason
+            if reason == LXMF.LXMessage.SOURCE_UNKNOWN:
                 return "Unknown Origin"
-            elif self.lxm.unverified_reason == LXMF.LXMessage.SIGNATURE_INVALID:
+            elif reason == LXMF.LXMessage.SIGNATURE_INVALID:
                 return "Invalid Signature"
             else:
                 return "Unknown signature validation failure"
+
+    def get_fields(self):
+        if not self.loaded:
+            self.load()
+
+        if self.lxm and hasattr(self.lxm, "get_fields"):
+            fields = self.lxm.get_fields()
+            if fields and isinstance(fields, dict):
+                return fields
+
+        return {}
+
+    def content_renderer(self):
+        if self._cached_renderer is not ConversationMessage._RENDERER_UNSET:
+            return self._cached_renderer
+        if not self.loaded:
+            self.load()
+        # load() sets _cached_renderer on success; if it failed, lxm is None
+        if self._cached_renderer is ConversationMessage._RENDERER_UNSET:
+            return None
+        return self._cached_renderer
+
+    def has_attachments(self):
+        if self._cached_has_attachments is not None:
+            return self._cached_has_attachments
+        fields = self.get_fields()
+        return (
+            LXMF.FIELD_FILE_ATTACHMENTS in fields
+            or LXMF.FIELD_IMAGE in fields
+            or LXMF.FIELD_AUDIO in fields
+        )
+
+    def get_file_attachments(self):
+        att_dir = self._attachment_dir()
+        if att_dir and os.path.isdir(att_dir):
+            manifest = self._read_attachment_manifest(att_dir)
+            if manifest and "files" in manifest:
+                result = []
+                for entry in manifest["files"]:
+                    fpath = os.path.join(att_dir, entry["stored_name"])
+                    if os.path.isfile(fpath):
+                        with open(fpath, "rb") as f:
+                            result.append([entry["name"], f.read()])
+                return result
+
+        fields = self.get_fields()
+        return fields.get(LXMF.FIELD_FILE_ATTACHMENTS, [])
+
+    def get_image(self):
+        att_dir = self._attachment_dir()
+        if att_dir and os.path.isdir(att_dir):
+            fpath = os.path.join(att_dir, "image")
+            if os.path.isfile(fpath):
+                with open(fpath, "rb") as f:
+                    return f.read()
+
+        fields = self.get_fields()
+        return fields.get(LXMF.FIELD_IMAGE, None)
+
+    def get_audio(self):
+        att_dir = self._attachment_dir()
+        if att_dir and os.path.isdir(att_dir):
+            fpath = os.path.join(att_dir, "audio")
+            if os.path.isfile(fpath):
+                with open(fpath, "rb") as f:
+                    return f.read()
+
+        fields = self.get_fields()
+        return fields.get(LXMF.FIELD_AUDIO, None)
+
+    def get_attachment_file_path(self, field_type, field_index=0):
+        att_dir = self._attachment_dir()
+        if att_dir and os.path.isdir(att_dir):
+            manifest = self._read_attachment_manifest(att_dir)
+            if manifest and "files" in manifest and field_index < len(manifest["files"]):
+                return os.path.join(att_dir, manifest["files"][field_index]["stored_name"])
+            # Fallback for old extraction format
+            if field_type == "image":
+                fpath = os.path.join(att_dir, "image")
+                if os.path.isfile(fpath):
+                    return fpath
+            elif field_type == "audio":
+                fpath = os.path.join(att_dir, "audio")
+                if os.path.isfile(fpath):
+                    return fpath
+        return None
+
+    def _attachment_dir(self):
+        try:
+            app = nomadnet.NomadNetworkApp.get_shared_instance()
+            msg_hash = self.get_hash()
+            if msg_hash:
+                return os.path.join(app.attachmentpath, RNS.hexrep(msg_hash, delimit=False))
+        except Exception:
+            pass
+        return None
+
+    def _read_attachment_manifest(self, att_dir):
+        manifest_path = os.path.join(att_dir, "manifest")
+        if os.path.isfile(manifest_path):
+            try:
+                with open(manifest_path, "rb") as f:
+                    manifest = msgpack.unpackb(f.read(), raw=False)
+                    safe_files = []
+                    for idx, entry in enumerate(manifest.get("files", [])):
+                        if not isinstance(entry, dict):
+                            continue
+                        entry["name"] = ConversationMessage.safe_attachment_name(entry.get("name"), fallback="attachment_"+str(idx))
+                        stored = entry.get("stored_name")
+                        if not isinstance(stored, str) or not re.fullmatch(r"file_\d+", stored):
+                            continue
+                        safe_files.append(entry)
+                    manifest["files"] = safe_files
+                    return manifest
+
+            except Exception as e:
+                RNS.log(f"Error loading attachment manifest: {e}")
+                return None
+
+        return None
+
+    @staticmethod
+    def extract_attachments_from_lxm(lxmessage, app):
+        if not hasattr(lxmessage, "get_fields"):
+            return
+        fields = lxmessage.get_fields()
+        if not fields or not isinstance(fields, dict):
+            return
+
+        has_any = (
+            LXMF.FIELD_FILE_ATTACHMENTS in fields
+            or LXMF.FIELD_IMAGE in fields
+            or LXMF.FIELD_AUDIO in fields
+        )
+        if not has_any:
+            return
+
+        msg_hash_hex = RNS.hexrep(lxmessage.hash, delimit=False)
+        att_dir = os.path.join(app.attachmentpath, msg_hash_hex)
+        if os.path.isdir(att_dir):
+            return
+
+        os.makedirs(att_dir)
+        manifest = {"files": []}
+
+        file_attachments = fields.get(LXMF.FIELD_FILE_ATTACHMENTS, [])
+        if file_attachments:
+            for idx, att in enumerate(file_attachments):
+                try:
+                    if isinstance(att, list) and len(att) >= 2:
+                        filename = ConversationMessage.safe_attachment_name(att[0], fallback="attachment_"+str(idx))
+                        data = att[1] if isinstance(att[1], bytes) else b""
+                        stored_name = "file_"+str(idx)
+                        with open(os.path.join(att_dir, stored_name), "wb") as f: f.write(data)
+                        manifest["files"].append({"name": filename, "stored_name": stored_name, "size": len(data)})
+
+                except Exception as e:
+                    RNS.log(f"Error decoding file attachment: {e}", RNS.LOG_ERROR)
+                    continue
+
+        if LXMF.FIELD_IMAGE in fields:
+            fmt, data = ConversationMessage._unpack_media_field(fields[LXMF.FIELD_IMAGE])
+            if data:
+                ext = ConversationMessage._ext_from_media_format(fmt, data)
+                filename = ConversationMessage.safe_attachment_name("image" + ext, fallback="image")
+                stored_name = "file_"+str(len(manifest["files"]))
+                with open(os.path.join(att_dir, stored_name), "wb") as f:
+                    f.write(data)
+                manifest["files"].append({"name": filename, "stored_name": stored_name, "size": len(data)})
+
+        if LXMF.FIELD_AUDIO in fields:
+            fmt, data = ConversationMessage._unpack_media_field(fields[LXMF.FIELD_AUDIO])
+            if data:
+                ext = ConversationMessage._ext_from_media_format(fmt, data, is_audio=True)
+                filename = ConversationMessage.safe_attachment_name("audio" + ext, fallback="audio")
+                stored_name = "file_"+str(len(manifest["files"]))
+                with open(os.path.join(att_dir, stored_name), "wb") as f:
+                    f.write(data)
+                manifest["files"].append({"name": filename, "stored_name": stored_name, "size": len(data)})
+
+        with open(os.path.join(att_dir, "manifest"), "wb") as f:
+            f.write(msgpack.packb(manifest))
+
+    @staticmethod
+    def safe_attachment_name(name, fallback="attachment"):
+        try:
+            if isinstance(name, bytes):
+                name = name.decode("utf-8", errors="replace")
+            elif not isinstance(name, str):
+                name = str(name) if name is not None else ""
+        except Exception:
+            name = ""
+        name = re.sub(r"[\x00-\x1f\x7f-\x9f]", "", name)
+        parts = re.split(r"[/\\]", name)
+        name = parts[-1] if parts else ""
+        if ":" in name:
+            name = name.split(":")[-1]
+        name = name.lstrip(".")
+        if not name or name in (".", ".."):
+            return fallback
+        if len(name) > 200:
+            base, ext = os.path.splitext(name)
+            ext = ext[:16]
+            name = base[:200 - len(ext)] + ext
+        return name
+
+    @staticmethod
+    def _unpack_media_field(field_data):
+        """Normalize FIELD_IMAGE/FIELD_AUDIO which can be raw bytes or [format, bytes].
+        Format element can be a string ('webp') or integer (LXMF audio mode constant)."""
+        if isinstance(field_data, bytes):
+            return None, field_data
+        elif isinstance(field_data, list) and len(field_data) >= 2 and isinstance(field_data[1], bytes):
+            return field_data[0], field_data[1]
+        return None, None
+
+    @staticmethod
+    def _detect_image_ext(data):
+        if not isinstance(data, bytes) or len(data) < 12:
+            return ".bin"
+        if data[:8] == b'\x89PNG\r\n\x1a\n':
+            return ".png"
+        elif data[:3] == b'\xff\xd8\xff':
+            return ".jpg"
+        elif data[:4] == b'GIF8':
+            return ".gif"
+        elif data[:4] == b'RIFF' and data[8:12] == b'WEBP':
+            return ".webp"
+        elif data[:4] == b'\x00\x00\x00\x1c' or data[:4] == b'\x00\x00\x00\x18':
+            return ".heic"
+        return ".bin"
+
+    @staticmethod
+    def _detect_audio_ext(data):
+        if not isinstance(data, bytes) or len(data) < 12:
+            return ".bin"
+        if data[:4] == b'OggS':
+            return ".ogg"
+        elif data[:2] == b'\xff\xfb' or data[:3] == b'ID3':
+            return ".mp3"
+        elif data[:4] == b'RIFF' and data[8:12] == b'WAVE':
+            return ".wav"
+        elif data[:4] == b'fLaC':
+            return ".flac"
+        return ".bin"
+
+    @staticmethod
+    def _ext_from_media_format(fmt, data, is_audio=False):
+        if isinstance(fmt, str) and len(fmt) > 0:
+            safe = re.sub(r"[^A-Za-z0-9]", "", fmt).lower()[:8]
+            if safe:
+                return "." + safe
+        if isinstance(fmt, int) and is_audio:
+            if fmt >= 16 and fmt <= 25:
+                return ".ogg"
+            elif fmt >= 1 and fmt <= 9:
+                return ".c2"
+        if is_audio:
+            return ConversationMessage._detect_audio_ext(data)
+        return ConversationMessage._detect_image_ext(data)
+
+    @staticmethod
+    def strip_attachments_from_file(file_path, app):
+        try:
+            with open(file_path, "rb") as f:
+                container = msgpack.unpackb(f.read(), strict_map_key=False)
+
+            lxmf_bytes = container[b"lxmf_bytes"] if b"lxmf_bytes" in container else container.get("lxmf_bytes")
+            if lxmf_bytes is None:
+                return
+
+            dest_len = LXMF.LXMessage.DESTINATION_LENGTH
+            sig_len  = LXMF.LXMessage.SIGNATURE_LENGTH
+            header_len = 2 * dest_len + sig_len
+
+            header = lxmf_bytes[:header_len]
+            payload_bytes = lxmf_bytes[header_len:]
+            payload = msgpack.unpackb(payload_bytes, strict_map_key=False)
+
+            if len(payload) < 4 or not isinstance(payload[3], dict):
+                return
+
+            fields = payload[3]
+            attachment_keys = [LXMF.FIELD_FILE_ATTACHMENTS, LXMF.FIELD_IMAGE, LXMF.FIELD_AUDIO]
+            if not any(k in fields for k in attachment_keys):
+                return
+
+            # Compute message hash matching LXMF's logic
+            if len(payload) > 4:
+                hash_payload = msgpack.packb(payload[:4])
+            else:
+                hash_payload = payload_bytes
+            msg_hash = RNS.Identity.full_hash(lxmf_bytes[:2*dest_len] + hash_payload)
+            msg_hash_hex = RNS.hexrep(msg_hash, delimit=False)
+
+            att_dir = os.path.join(app.attachmentpath, msg_hash_hex)
+            if not os.path.isdir(att_dir):
+                return
+
+            for k in attachment_keys:
+                if k in fields:
+                    del fields[k]
+
+            new_lxmf_bytes = header + msgpack.packb(payload)
+            key = b"lxmf_bytes" if b"lxmf_bytes" in container else "lxmf_bytes"
+            container[key] = new_lxmf_bytes
+
+            with open(file_path, "wb") as f:
+                f.write(msgpack.packb(container))
+
+        except Exception as e:
+            RNS.log("Error stripping attachments from LXM file: "+str(e), RNS.LOG_ERROR)
+
+    def to_index_entry(self):
+        return {
+            "timestamp": self.timestamp,
+            "sort_timestamp": self.sort_timestamp,
+            "state": self._cached_state,
+            "title": self._cached_title,
+            "content": self._cached_content,
+            "source_hash": self._cached_source_hash,
+            "transport_encrypted": self._cached_transport_encrypted,
+            "transport_encryption": self._cached_transport_encryption,
+            "signature_validated": self._cached_signature_validated,
+            "unverified_reason": self._cached_unverified_reason,
+            "method": self._cached_method,
+            "renderer": None if self._cached_renderer is ConversationMessage._RENDERER_UNSET else self._cached_renderer,
+            "has_attachments": self._cached_has_attachments,
+            "attachment_names": self._cached_attachment_names,
+            "rssi": self._cached_rssi,
+            "snr": self._cached_snr,
+            "q": self._cached_q,
+        }
+
+    def restore_from_index(self, entry):
+        self.timestamp = entry.get("timestamp")
+        self.sort_timestamp = entry.get("sort_timestamp", self.sort_timestamp)
+        self._cached_state = entry.get("state")
+        self._cached_title = entry.get("title")
+        self._cached_content = entry.get("content")
+        self._cached_source_hash = entry.get("source_hash")
+        self._cached_transport_encrypted = entry.get("transport_encrypted")
+        self._cached_transport_encryption = entry.get("transport_encryption")
+        self._cached_signature_validated = entry.get("signature_validated")
+        self._cached_unverified_reason = entry.get("unverified_reason")
+        self._cached_method = entry.get("method")
+        self._cached_renderer = entry.get("renderer", ConversationMessage._RENDERER_UNSET)
+        self._cached_has_attachments = entry.get("has_attachments")
+        self._cached_attachment_names = entry.get("attachment_names")
+        self._cached_rssi = entry.get("rssi")
+        self._cached_snr = entry.get("snr")
+        self._cached_q = entry.get("q")
+
+    @staticmethod
+    def read_index(conversation_path):
+        index_path = os.path.join(conversation_path, ".index")
+        if os.path.isfile(index_path):
+            try:
+                with open(index_path, "rb") as f:
+                    return msgpack.unpackb(f.read(), raw=False)
+            except Exception:
+                pass
+        return {}
+
+    @staticmethod
+    def write_index(conversation_path, messages):
+        index_path = os.path.join(conversation_path, ".index")
+        index = {}
+        if os.path.isfile(index_path):
+            try:
+                with open(index_path, "rb") as f:
+                    index = msgpack.unpackb(f.read(), raw=False)
+            except Exception:
+                index = {}
+
+        for msg in messages:
+            if msg._cached_state is not None:
+                filename = os.path.basename(msg.file_path)
+                index[filename] = msg.to_index_entry()
+
+        try:
+            with open(index_path, "wb") as f:
+                f.write(msgpack.packb(index))
+        except Exception as e:
+            RNS.log("Error writing conversation index: "+str(e), RNS.LOG_ERROR)

nomadnet/Directory.py

@@ -6,7 +6,9 @@ import nomadnet
 import threading
 import RNS.vendor.umsgpack as msgpack
 
-from LXMF import pn_announce_data_is_valid
+from LXMF import pn_announce_data_is_valid, PN_META_IMPL_NAME, PN_META_VERSION
+from nomadnet.util import strip_modifiers
+from nomadnet.util import sanitize_name
 
 class PNAnnounceHandler:
     def __init__(self, owner):
@@ -19,7 +21,10 @@ class PNAnnounceHandler:
                 data = msgpack.unpackb(app_data)
 
                 if data[2] == True:
-                    RNS.log("Received active propagation node announce from "+RNS.prettyhexrep(destination_hash))
+                    metadata = data[6]
+                    if not PN_META_IMPL_NAME in metadata or not PN_META_VERSION in metadata: impl_str = ""
+                    else: impl_str = f" running {metadata[PN_META_IMPL_NAME]} {metadata[PN_META_VERSION]}"
+                    RNS.log(f"Received active propagation node announce from {RNS.prettyhexrep(destination_hash)}{impl_str}", RNS.LOG_DEBUG)
 
                     associated_peer = RNS.Destination.hash_from_name_and_identity("lxmf.delivery", announced_identity)
                     associated_node = RNS.Destination.hash_from_name_and_identity("nomadnetwork.node", announced_identity)
@@ -54,9 +59,27 @@ class Directory:
             RNS.log("The contained exception was: "+str(e), RNS.LOG_DEBUG)
 
 
+    @property
+    def announce_stream(self):
+        return self._node_announces+self._peer_announces+self._pn_announces
+
+    def _clean_node_announces(self):
+        while len(self._node_announces) > Directory.ANNOUNCE_STREAM_MAXLENGTH:
+            self._node_announces.pop()
+
+    def _clean_peer_announces(self):
+        while len(self._peer_announces) > Directory.ANNOUNCE_STREAM_MAXLENGTH:
+            self._peer_announces.pop()
+
+    def _clean_pn_announces(self):
+        while len(self._pn_announces) > Directory.ANNOUNCE_STREAM_MAXLENGTH:
+            self._pn_announces.pop()
+
     def __init__(self, app):
         self.directory_entries = {}
-        self.announce_stream = []
+        self._node_announces = []
+        self._peer_announces = []
+        self._pn_announces   = []
         self.app = app
         self.announce_lock = threading.Lock()
         self.load_from_disk()
@@ -70,16 +93,15 @@ class Directory:
             packed_list = []
             for source_hash in self.directory_entries:
                 e = self.directory_entries[source_hash]
-                packed_list.append((e.source_hash, e.display_name, e.trust_level, e.hosts_node, e.preferred_delivery, e.identify, e.sort_rank))
+                packed_list.append((e.source_hash, e.display_name, e.trust_level, e.hosts_node, e.preferred_delivery, e.identify, e.sort_rank, getattr(e, "notes", "")))
 
             directory = {
                 "entry_list": packed_list,
                 "announce_stream": self.announce_stream
             }
 
-            file = open(self.app.directorypath, "wb")
+            with open(self.app.directorypath, "wb") as file:
-            file.write(msgpack.packb(directory))
+                file.write(msgpack.packb(directory))
-            file.close()
 
         except Exception as e:
             RNS.log("Could not write directory to disk. Then contained exception was: "+str(e), RNS.LOG_ERROR)
@@ -87,10 +109,9 @@ class Directory:
     def load_from_disk(self):
         if os.path.isfile(self.app.directorypath):
             try:
-                file = open(self.app.directorypath, "rb")
+                with open(self.app.directorypath, "rb") as file:
-                unpacked_directory = msgpack.unpackb(file.read())
+                    unpacked_directory = msgpack.unpackb(file.read())
                 unpacked_list = unpacked_directory["entry_list"]
-                file.close()
 
                 entries = {}
                 for e in unpacked_list:
@@ -118,11 +139,19 @@ class Directory:
                     else:
                         sort_rank = None
 
-                    entries[e[0]] = DirectoryEntry(e[0], e[1], e[2], hosts_node, preferred_delivery=preferred_delivery, identify_on_connect=identify, sort_rank=sort_rank)
+                    if len(e) > 7:
+                        notes = e[7]
+                    else:
+                        notes = None
+
+                    entries[e[0]] = DirectoryEntry(e[0], e[1], e[2], hosts_node, preferred_delivery=preferred_delivery, identify_on_connect=identify, sort_rank=sort_rank, notes=notes)
 
                 self.directory_entries = entries
 
-                self.announce_stream = unpacked_directory["announce_stream"]
+                astream = unpacked_directory["announce_stream"]
+                self._node_announces = [e for e in astream if e[3] == "node"]
+                self._peer_announces = [e for e in astream if e[3] == "peer"]
+                self._pn_announces = [e for e in astream if e[3] == "pn"]
 
             except Exception as e:
                 RNS.log("Could not load directory from disk. The contained exception was: "+str(e), RNS.LOG_ERROR)
@@ -133,20 +162,17 @@ class Directory:
                 if self.app.compact_stream:
                     try:
                         remove_announces = []
-                        for announce in self.announce_stream:
+                        for announce in self._peer_announces:
-                            if announce[1] == source_hash:
+                            if announce[1] == source_hash: remove_announces.append(announce)
-                                remove_announces.append(announce)
 
-                        for a in remove_announces:
+                        for a in remove_announces: self._peer_announces.remove(a)
-                            self.announce_stream.remove(a)
                     
                     except Exception as e:
                         RNS.log("An error occurred while compacting the announce stream. The contained exception was:"+str(e), RNS.LOG_ERROR)
 
                 timestamp = time.time()
-                self.announce_stream.insert(0, (timestamp, source_hash, app_data, "peer"))
+                self._peer_announces.insert(0, (timestamp, source_hash, app_data, "peer"))
-                while len(self.announce_stream) > Directory.ANNOUNCE_STREAM_MAXLENGTH:
+                self._clean_peer_announces()
-                    self.announce_stream.pop()
 
                 if hasattr(self.app, "ui") and self.app.ui != None:
                     if hasattr(self.app.ui, "main_display"):
@@ -158,26 +184,25 @@ class Directory:
                 if self.app.compact_stream:
                     try:
                         remove_announces = []
-                        for announce in self.announce_stream:
+                        for announce in self._node_announces:
                             if announce[1] == source_hash:
                                 remove_announces.append(announce)
 
                         for a in remove_announces:
-                            self.announce_stream.remove(a)
+                            self._node_announces.remove(a)
                     
                     except Exception as e:
                         RNS.log("An error occurred while compacting the announce stream. The contained exception was:"+str(e), RNS.LOG_ERROR)
 
                 timestamp = time.time()
-                self.announce_stream.insert(0, (timestamp, source_hash, app_data, "node"))
+                self._node_announces.insert(0, (timestamp, source_hash, app_data, "node"))
-                while len(self.announce_stream) > Directory.ANNOUNCE_STREAM_MAXLENGTH:
+                self._clean_node_announces()
-                    self.announce_stream.pop()
 
-                if self.trust_level(associated_peer) == DirectoryEntry.TRUSTED:
+                # if self.trust_level(associated_peer) == DirectoryEntry.TRUSTED:
-                    existing_entry = self.find(source_hash)
+                #     existing_entry = self.find(source_hash)
-                    if not existing_entry:
+                #     if not existing_entry:
-                        node_entry = DirectoryEntry(source_hash, display_name=app_data.decode("utf-8"), trust_level=DirectoryEntry.TRUSTED, hosts_node=True)
+                #         node_entry = DirectoryEntry(source_hash, display_name=app_data.decode("utf-8"), trust_level=DirectoryEntry.TRUSTED, hosts_node=True)
-                        self.remember(node_entry)
+                #         self.remember(node_entry)
                 
                 if hasattr(self.app.ui, "main_display"):
                     self.app.ui.main_display.sub_displays.network_display.directory_change_callback()
@@ -190,7 +215,7 @@ class Directory:
                     found_node = True
                     break
 
-            for e in self.announce_stream:
+            for e in self._pn_announces:
                 if e[1] == associated_node:
                     found_node = True
                     break
@@ -200,63 +225,83 @@ class Directory:
                 if self.app.compact_stream:
                     try:
                         remove_announces = []
-                        for announce in self.announce_stream:
+                        for announce in self._pn_announces:
                             if announce[1] == source_hash:
                                 remove_announces.append(announce)
 
                         for a in remove_announces:
-                            self.announce_stream.remove(a)
+                            self._pn_announces.remove(a)
                     
                     except Exception as e:
                         RNS.log("An error occurred while compacting the announce stream. The contained exception was:"+str(e), RNS.LOG_ERROR)
 
                 timestamp = time.time()
-                self.announce_stream.insert(0, (timestamp, source_hash, app_data, "pn"))
+                self._pn_announces.insert(0, (timestamp, source_hash, app_data, "pn"))
-                while len(self.announce_stream) > Directory.ANNOUNCE_STREAM_MAXLENGTH:
+                self._clean_pn_announces()
-                    self.announce_stream.pop()
                 
                 if hasattr(self.app, "ui") and hasattr(self.app.ui, "main_display"):
                     self.app.ui.main_display.sub_displays.network_display.directory_change_callback()
 
     def remove_announce_with_timestamp(self, timestamp):
         selected_announce = None
-        for announce in self.announce_stream:
+        for announce in self._node_announces:
             if announce[0] == timestamp:
                 selected_announce = announce
+                break
 
         if selected_announce != None:
-            self.announce_stream.remove(selected_announce)
+            self._node_announces.remove(selected_announce)
+            return
+
+        for announce in self._peer_announces:
+            if announce[0] == timestamp:
+                selected_announce = announce
+                break
+
+        if selected_announce != None:
+            self._peer_announces.remove(selected_announce)
+            return
+
+        for announce in self._pn_announces:
+            if announce[0] == timestamp:
+                selected_announce = announce
+                break
+
+        if selected_announce != None:
+            self._pn_announces.remove(selected_announce)
+            return
 
     def display_name(self, source_hash):
         if source_hash in self.directory_entries:
-            return self.directory_entries[source_hash].display_name
+            return strip_modifiers(self.directory_entries[source_hash].display_name)
         else:
             return None
 
-    def simplest_display_str(self, source_hash):
+    def simplest_display_str(self, source_hash, san=True):
+        def s(name):
+            if self.app.config["textui"]["sanitize_names"] and san: return sanitize_name(name)
+            else:                                                   return strip_modifiers(name)
+
         trust_level = self.trust_level(source_hash)
         if trust_level == DirectoryEntry.WARNING or trust_level == DirectoryEntry.UNTRUSTED:
             if source_hash in self.directory_entries:
-                dn = self.directory_entries[source_hash].display_name
+                dn = s(self.directory_entries[source_hash].display_name)
-                if dn == None:
+                if not dn: return RNS.prettyhexrep(source_hash)
-                    return RNS.prettyhexrep(source_hash)
+                else:      return dn+" <"+RNS.hexrep(source_hash, delimit=False)+">"
-                else:
+            
-                    return dn+" <"+RNS.hexrep(source_hash, delimit=False)+">"
+            else: return "<"+RNS.hexrep(source_hash, delimit=False)+">"
-            else:
+        
-                return "<"+RNS.hexrep(source_hash, delimit=False)+">"
         else:
             if source_hash in self.directory_entries:
-                dn = self.directory_entries[source_hash].display_name
+                dn = s(self.directory_entries[source_hash].display_name)
-                if dn == None:
+                if not dn: return RNS.prettyhexrep(source_hash)
-                    return RNS.prettyhexrep(source_hash)
+                else:      return dn
-                else:
+            
-                    return dn
+            else: return "<"+RNS.hexrep(source_hash, delimit=False)+">"
-            else:
-                return "<"+RNS.hexrep(source_hash, delimit=False)+">"
 
     def alleged_display_str(self, source_hash):
         if source_hash in self.directory_entries:
-            return self.directory_entries[source_hash].display_name
+            return strip_modifiers(self.directory_entries[source_hash].display_name)
         else:
             return None
 
@@ -370,10 +415,11 @@ class DirectoryEntry:
     UNKNOWN   = 0x02
     TRUSTED   = 0xFF
 
-    DIRECT     = 0x01
+    DIRECT        = 0x01
-    PROPAGATED = 0x02
+    PROPAGATED    = 0x02
+    OPPORTUNISTIC = 0x03
 
-    def __init__(self, source_hash, display_name=None, trust_level=UNKNOWN, hosts_node=False, preferred_delivery=None, identify_on_connect=False, sort_rank=None):
+    def __init__(self, source_hash, display_name=None, trust_level=UNKNOWN, hosts_node=False, preferred_delivery=None, identify_on_connect=False, sort_rank=None, notes=None):
         if len(source_hash) == RNS.Identity.TRUNCATED_HASHLENGTH//8:
             self.source_hash  = source_hash
             self.display_name = display_name
@@ -387,5 +433,6 @@ class DirectoryEntry:
             self.trust_level  = trust_level
             self.hosts_node   = hosts_node
             self.identify     = identify_on_connect
+            self.notes        = notes or ""
         else:
             raise TypeError("Attempt to add invalid source hash to directory")

nomadnet/InterfaceProfiles.py

@@ -0,0 +1,240 @@
+import os
+import threading
+
+import RNS
+import RNS.vendor.umsgpack as msgpack
+from nomadnet.util import write_config_atomic
+
+
+class InterfaceProfiles:
+    VERSION = 1
+
+    def __init__(self, app):
+        self.app = app
+        self.path = os.path.join(app.storagepath, "interface_profiles.msgpack")
+        self._lock = threading.Lock()
+        self.profiles = []
+        self.load()
+
+    def load(self):
+        self.profiles = []
+        self.default_members = None
+        try:
+            if not os.path.isfile(self.path):
+                return
+            with open(self.path, "rb") as f:
+                data = msgpack.unpackb(f.read())
+            if not isinstance(data, dict):
+                return
+            clean = []
+            seen = set()
+            for p in data.get("profiles") or []:
+                if not isinstance(p, dict):
+                    continue
+                pid = str(p.get("id") or "")
+                if not pid or pid in seen:
+                    continue
+                members = [str(m) for m in (p.get("members") or []) if isinstance(m, (str, bytes))]
+                clean.append({"id": pid, "name": str(p.get("name") or ""), "members": members})
+                seen.add(pid)
+            self.profiles = clean
+            dm = data.get("default_members")
+            self.default_members = [str(m) for m in dm] if isinstance(dm, list) else None
+        except Exception as e:
+            RNS.log("Could not load interface profiles: "+str(e), RNS.LOG_ERROR)
+            self.profiles = []
+            self.default_members = None
+
+    def save(self):
+        with self._lock:
+            tmp = self.path + ".tmp"
+            try:
+                payload = {
+                    "version": InterfaceProfiles.VERSION,
+                    "profiles": [{"id": p["id"], "name": p["name"], "members": list(p["members"])} for p in self.profiles],
+                    "default_members": self.default_members,
+                }
+                with open(tmp, "wb") as f:
+                    f.write(msgpack.packb(payload))
+                    f.flush()
+                    try: os.fsync(f.fileno())
+                    except Exception: pass
+                os.replace(tmp, self.path)
+            except Exception as e:
+                RNS.log("Could not save interface profiles: "+str(e), RNS.LOG_ERROR)
+                try: os.remove(tmp)
+                except Exception: pass
+
+    def _interfaces(self):
+        try:
+            return self.app.rns.config["interfaces"]
+        except Exception:
+            return {}
+
+    def interface_names(self):
+        try:
+            return list(self._interfaces().keys())
+        except Exception:
+            return []
+
+    def _is_enabled(self, iface):
+        return str(iface.get("enabled")).lower() not in ('false', 'off', 'no', '0') and \
+               str(iface.get("interface_enabled")).lower() not in ('false', 'off', 'no', '0')
+
+    def enabled_set(self):
+        interfaces = self._interfaces()
+        result = set()
+        for name in interfaces:
+            try:
+                if self._is_enabled(interfaces[name]):
+                    result.add(name)
+            except Exception:
+                pass
+        return result
+
+    def get(self, pid):
+        for p in self.profiles:
+            if p["id"] == pid:
+                return p
+        return None
+
+    def members_existing(self, p):
+        existing = set(self.interface_names())
+        return set(m for m in p["members"] if m in existing)
+
+    def active_profile_id(self):
+        enabled = self.enabled_set()
+        for p in self.profiles:
+            if self.members_existing(p) == enabled:
+                return p["id"]
+        return None
+
+    def profiles_for(self, iface_name):
+        return [p for p in self.profiles if iface_name in p["members"]]
+
+    def label_for(self, iface_name):
+        names = [p["name"] or p["id"] for p in self.profiles_for(iface_name)]
+        return ", ".join(names) if names else None
+
+    def member_count(self, p):
+        return len(self.members_existing(p))
+
+    def _new_id(self):
+        while True:
+            pid = os.urandom(4).hex()
+            if not self.get(pid):
+                return pid
+
+    def create(self, name):
+        pid = self._new_id()
+        self.profiles.append({"id": pid, "name": name or "Profile", "members": []})
+        self.save()
+        return pid
+
+    def save_current_as_profile(self, name):
+        pid = self.create(name)
+        self.set_members(pid, sorted(self.enabled_set()))
+        return pid
+
+    def rename(self, pid, name):
+        p = self.get(pid)
+        if p is not None and name:
+            p["name"] = name
+            self.save()
+
+    def delete(self, pid):
+        self.profiles = [p for p in self.profiles if p["id"] != pid]
+        self.save()
+
+    def set_members(self, pid, members):
+        p = self.get(pid)
+        if p is not None:
+            valid = set(self.interface_names())
+            seen = set()
+            ordered = []
+            for m in members:
+                if m in valid and m not in seen:
+                    ordered.append(m)
+                    seen.add(m)
+            p["members"] = ordered
+            self.save()
+
+    def set_interface_profiles(self, iface_name, profile_ids):
+        pset = set(profile_ids)
+        for p in self.profiles:
+            if p["id"] in pset:
+                if iface_name not in p["members"]:
+                    p["members"].append(iface_name)
+            elif iface_name in p["members"]:
+                p["members"] = [m for m in p["members"] if m != iface_name]
+        self.save()
+
+    def _apply_members(self, members):
+        existing = set(self.interface_names())
+        members = set(m for m in members if m in existing)
+        interfaces = self._interfaces()
+        try:
+            for name in interfaces:
+                value = "true" if name in members else "false"
+                interfaces[name]["interface_enabled"] = value
+                if "enabled" in interfaces[name]:
+                    interfaces[name]["enabled"] = value
+            write_config_atomic(self.app.rns.config)
+            return True
+        except Exception as e:
+            RNS.log("Could not apply interface profile: "+str(e), RNS.LOG_ERROR)
+            return False
+
+    def select_profile(self, pid):
+        p = self.get(pid)
+        if p is None:
+            return False
+        return self._apply_members(self.members_existing(p))
+
+    def select_default(self):
+        if self.default_members is None:
+            return False
+        return self._apply_members(set(self.default_members))
+
+    def update_default_if_custom(self):
+        # Remember the live enabled-set as "Default" whenever it matches no named
+        # profile -- i.e. it's the user's manual setup. Only relevant once profiles exist.
+        if not self.profiles:
+            return
+        enabled = self.enabled_set()
+        matches_named = any(self.members_existing(p) == enabled for p in self.profiles)
+        if matches_named and self.default_members is not None:
+            return
+        new_default = sorted(enabled)
+        if self.default_members != new_default:
+            self.default_members = new_default
+            self.save()
+
+    def remove_interface(self, iface_name):
+        changed = False
+        for p in self.profiles:
+            if iface_name in p["members"]:
+                p["members"] = [m for m in p["members"] if m != iface_name]
+                changed = True
+        if changed:
+            self.save()
+
+    def rename_interface(self, old_name, new_name):
+        changed = False
+        for p in self.profiles:
+            if old_name in p["members"]:
+                p["members"] = [new_name if m == old_name else m for m in p["members"]]
+                changed = True
+        if changed:
+            self.save()
+
+    def prune(self):
+        valid = set(self.interface_names())
+        changed = False
+        for p in self.profiles:
+            pruned = [m for m in p["members"] if m in valid]
+            if len(pruned) != len(p["members"]):
+                p["members"] = pruned
+                changed = True
+        if changed:
+            self.save()

nomadnet/Node.py

@@ -4,11 +4,16 @@ import sys
 import RNS
 import time
 import threading
+import hashlib
+import shutil
 import subprocess
 import RNS.vendor.umsgpack as msgpack
 
+from RNS.Utilities.rngit.media import convert_file_to_webp
+
 class Node:
     JOB_INTERVAL = 5
+    SETTINGS_FLUSH_INTERVAL = 60
     START_ANNOUNCE_DELAY = 6
 
     def __init__(self, app):
@@ -24,11 +29,15 @@ class Node:
         self.file_refresh_interval = self.app.file_refresh_interval
         self.job_interval = Node.JOB_INTERVAL
         self.should_run_jobs = True
+        self.settings_dirty = False
+        self.conversion_cache_lock = threading.Lock()
+        self.last_settings_flush = time.time()
         self.app_data = None
         self.name = self.app.node_name
 
         self.register_pages()
         self.register_files()
+        self.register_media()
 
         self.destination.set_link_established_callback(self.peer_connected)
 
@@ -70,6 +79,10 @@ class Node:
                 response_generator = self.serve_page,
                 allow = RNS.Destination.ALLOW_ALL)
 
+    def register_media(self):
+        self.destination.register_request_handler("/media", response_generator = self.serve_media,
+                                                  allow = RNS.Destination.ALLOW_ALL)
+
     def register_files(self):
         # TODO: Deregister previously registered files
         # that no longer exist.
@@ -100,7 +113,8 @@ class Node:
         directories = [file for file in os.listdir(base_path) if os.path.isdir(os.path.join(base_path, file)) and file[:1] != "."]
 
         for file in files:
-            self.servedfiles.append(base_path+"/"+file)
+            if not file.endswith(".allowed"):
+                self.servedfiles.append(base_path+"/"+file)
 
         for directory in directories:
             self.scan_files(base_path+"/"+directory)
@@ -109,52 +123,13 @@ class Node:
         RNS.log("Page request "+RNS.prettyhexrep(request_id)+" for: "+str(path), RNS.LOG_VERBOSE)
         try:
             self.app.peer_settings["served_page_requests"] += 1
-            self.app.save_peer_settings()
+            self.settings_dirty = True
             
         except Exception as e:
             RNS.log("Could not increase served page request count", RNS.LOG_ERROR)
 
         file_path = path.replace("/page", self.app.pagespath, 1)
-
+        request_allowed = self.request_allowed(file_path, remote_identity)
-        allowed_path = file_path+".allowed"
-        request_allowed = False
-
-        if os.path.isfile(allowed_path):
-            allowed_list = []
-
-            try:
-                if os.access(allowed_path, os.X_OK):
-                    allowed_result = subprocess.run([allowed_path], stdout=subprocess.PIPE)
-                    allowed_input = allowed_result.stdout
-
-                else:
-                    fh = open(allowed_path, "rb")
-                    allowed_input = fh.read()
-                    fh.close()
-
-                allowed_hash_strs = allowed_input.splitlines()
-
-                for hash_str in allowed_hash_strs:
-                    if len(hash_str) == RNS.Identity.TRUNCATED_HASHLENGTH//8*2:
-                        try:
-                            allowed_hash = bytes.fromhex(hash_str.decode("utf-8"))
-                            allowed_list.append(allowed_hash)
-
-                        except Exception as e:
-                            RNS.log("Could not decode RNS Identity hash from: "+str(hash_str), RNS.LOG_DEBUG)
-                            RNS.log("The contained exception was: "+str(e), RNS.LOG_DEBUG)
-
-            except Exception as e:
-                RNS.log("Error while fetching list of allowed identities for request: "+str(e), RNS.LOG_ERROR)
-
-            if hasattr(remote_identity, "hash") and remote_identity.hash in allowed_list:
-                request_allowed = True
-            else:
-                request_allowed = False
-                RNS.log("Denying request, remote identity was not in list of allowed identities", RNS.LOG_VERBOSE)
-
-        else:
-            request_allowed = True
 
         try:
             if request_allowed:
@@ -173,12 +148,11 @@ class Node:
                             if isinstance(e, str) and (e.startswith("field_") or e.startswith("var_")):
                                 env_map[e] = data[e]
 
-                    generated = subprocess.run([file_path], stdout=subprocess.PIPE, env=env_map)
+                    generated = subprocess.run([file_path], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, env=env_map)
                     return generated.stdout
                 else:
-                    fh = open(file_path, "rb")
+                    with open(file_path, "rb") as fh:
-                    response_data = fh.read()
+                        response_data = fh.read()
-                    fh.close()
                     return response_data
             else:
                 RNS.log("Request denied", RNS.LOG_VERBOSE)
@@ -189,18 +163,171 @@ class Node:
             RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
             return None
 
-    # TODO: Improve file handling, this will be slow for large files
+    NATIVE_MEDIA_EXTS = [".webp"]
+    MEDIA_EXTS = [".webp", ".png", ".jpg", ".jpeg", ".bmp", ".gif", ".tiff"]
+    CONVERSION_QUALITY = 85
+    CONVERSION_MAX_DIMENSION = 1200
+    CONVERSION_CACHE_MAX_FILES = 256
+    CONVERSION_CACHE_MAX_BYTES = 192*1024*1024
+    def serve_media(self, path, data, request_id, link_id, remote_identity, requested_at):
+        if not type(data) == dict: return None
+        if not "path" in data: return None
+        if not "key" in data: return None
+
+        jail       = os.path.normpath(self.app.pagespath)
+        media_path = os.path.join(self.app.pagespath, data["path"].replace("/media/", "").lstrip("/"))
+        media_path = os.path.normpath(media_path)
+        base_name  = os.path.basename(media_path)
+        base_ext   = os.path.splitext(base_name)[-1]
+
+        if not base_ext.lower() in self.MEDIA_EXTS:
+            RNS.log(f"Invalid media request type: {media_path}, must be in {self.MEDIA_EXTS}", RNS.LOG_DEBUG)
+            return False
+
+        if not media_path.startswith(jail+os.sep):
+            RNS.log(f"Invalid media request path: {media_path}", RNS.LOG_DEBUG)
+            return False
+
+        if len(media_path) > 512:
+            RNS.log(f"Invalid media request path length: {len(media_path)}", RNS.LOG_DEBUG)
+            return False
+
+        RNS.log(f"Media request {RNS.prettyhexrep(request_id)} for: {media_path}", RNS.LOG_VERBOSE)
+        try:
+            self.app.peer_settings["served_media_requests"] += 1
+            self.settings_dirty = True
+        except Exception as e: RNS.log("Could not increase served page request count", RNS.LOG_ERROR)
+
+        request_allowed = self.request_allowed(media_path, remote_identity)
+        try:
+            if request_allowed:
+                if not base_ext.lower() in self.NATIVE_MEDIA_EXTS:
+                    converted_path = self.convert_media_to_webp(media_path)
+                    if converted_path is False:
+                        RNS.log(f"Could not convert {media_path} to WebP", RNS.LOG_DEBUG)
+                        return False
+                    file_path = converted_path
+                    file_name = os.path.splitext(base_name)[0]+".webp"
+                else:
+                    file_path = media_path
+                    file_name = base_name
+
+                RNS.log(f"Serving media: {file_path}", RNS.LOG_VERBOSE)
+                return [open(file_path, "rb"), {"name": file_name.encode("utf-8")}]
+            
+            else:
+                RNS.log("Request denied", RNS.LOG_VERBOSE)
+                return False
+
+        except Exception as e:
+            RNS.log("Error occurred while handling request "+RNS.prettyhexrep(request_id)+" for: "+str(media_path), RNS.LOG_ERROR)
+            RNS.log("The contained exception was: "+str(e), RNS.LOG_ERROR)
+            return False
+
+    def convert_media_to_webp(self, media_path, quality = CONVERSION_QUALITY, max_dimension = CONVERSION_MAX_DIMENSION):
+        try:
+            hasher = hashlib.sha256()
+            with open(media_path, "rb") as fh:
+                for chunk in iter(lambda: fh.read(65536), b""): hasher.update(chunk)
+            source_hash = hasher.hexdigest()
+        except Exception as e:
+            RNS.log(f"Could not read media source for conversion: {media_path}", RNS.LOG_WARNING)
+            return False
+
+        if quality == None: quality = Node.CONVERSION_QUALITY
+        if max_dimension == None: max_dimension = Node.CONVERSION_MAX_DIMENSION
+
+        cache_name  = f"{source_hash}.q{quality}.d{max_dimension if max_dimension is not None else 0}.webp"
+        cache_path  = os.path.join(self.app.convcachepath, cache_name)
+        return self.__cached_conversion(media_path, cache_path, quality, max_dimension)
+
+    def __cached_conversion(self, media_path, cache_path, quality, max_dimension):
+        with self.conversion_cache_lock:
+            if os.path.isfile(cache_path):
+                RNS.log(f"Using cached media conversion for: {media_path}", RNS.LOG_VERBOSE)
+                try: os.utime(cache_path, None)
+                except Exception: pass
+                return cache_path
+
+            try:
+                os.makedirs(self.app.convcachepath, exist_ok=True)
+                converted_path = convert_file_to_webp(media_path, quality=quality, max_dimension=max_dimension)
+                if converted_path is False: return False
+
+                shutil.move(converted_path, cache_path)
+                self._prune_conversion_cache()
+                RNS.log(f"Converted media: {media_path} -> {cache_path}", RNS.LOG_VERBOSE)
+                return cache_path
+
+            except Exception as e:
+                RNS.log(f"Error during media conversion of {media_path}: {e}", RNS.LOG_WARNING)
+                return False
+
+    def _prune_conversion_cache(self):
+        try:
+            entries = []
+            for name in os.listdir(self.app.convcachepath):
+                path = os.path.join(self.app.convcachepath, name)
+                try:
+                    st = os.stat(path)
+                    if st.st_size > 0:
+                        entries.append((st.st_mtime, st.st_size, path))
+                except OSError: pass
+
+            total = sum(e[1] for e in entries)
+            entries.sort()
+            while len(entries) > Node.CONVERSION_CACHE_MAX_FILES or total > Node.CONVERSION_CACHE_MAX_BYTES:
+                _, size, path = entries.pop(0)
+                total -= size
+                try: os.unlink(path)
+                except OSError: pass
+        except OSError: pass
+
+    def request_allowed(self, file_path, remote_identity):
+        if file_path.lower().endswith(".allowed"): return False
+        allowed_path = file_path+".allowed"
+        if not os.path.isfile(allowed_path): return True
+
+        allowed_list = []
+        try:
+            if os.access(allowed_path, os.X_OK):
+                allowed_input = subprocess.run([allowed_path], stdout=subprocess.PIPE).stdout
+            else:
+                with open(allowed_path, "rb") as fh:
+                    allowed_input = fh.read()
+
+            for hash_str in allowed_input.splitlines():
+                if len(hash_str) == RNS.Identity.TRUNCATED_HASHLENGTH//8*2:
+                    try:
+                        allowed_list.append(bytes.fromhex(hash_str.decode("utf-8")))
+                    except Exception as e:
+                        RNS.log("Could not decode RNS Identity hash from: "+str(hash_str), RNS.LOG_DEBUG)
+                        RNS.log("The contained exception was: "+str(e), RNS.LOG_DEBUG)
+
+        except Exception as e:
+            RNS.log("Error while fetching list of allowed identities for request: "+str(e), RNS.LOG_ERROR)
+
+        if hasattr(remote_identity, "hash") and remote_identity.hash in allowed_list:
+            return True
+
+        RNS.log("Denying request, remote identity was not in list of allowed identities", RNS.LOG_VERBOSE)
+        return False
+
     def serve_file(self, path, data, request_id, remote_identity, requested_at):
         RNS.log("File request "+RNS.prettyhexrep(request_id)+" for: "+str(path), RNS.LOG_VERBOSE)
         try:
             self.app.peer_settings["served_file_requests"] += 1
-            self.app.save_peer_settings()
+            self.settings_dirty = True
             
         except Exception as e:
             RNS.log("Could not increase served file request count", RNS.LOG_ERROR)
 
         file_path = path.replace("/file", self.app.filespath, 1)
-        file_name = path.replace("/file/", "", 1)
+        file_name = os.path.basename(file_path)
+        if not self.request_allowed(file_path, remote_identity):
+            RNS.log("Request denied", RNS.LOG_VERBOSE)
+            return DEFAULT_NOTALLOWED.encode("utf-8")
+
         try:
             RNS.log("Serving file: "+file_path, RNS.LOG_VERBOSE)
             return [open(file_path, "rb"), {"name": file_name.encode("utf-8")}]
@@ -219,7 +346,7 @@ class Node:
         self.last_announce = time.time()
         self.app.peer_settings["node_last_announce"] = self.last_announce
         self.destination.announce(app_data=self.app_data)
-        self.app.message_router.announce_propagation_node()
+        if not self.app.disable_propagation: self.app.message_router.announce_propagation_node()
 
     def __jobs(self):
         while self.should_run_jobs:
@@ -238,13 +365,19 @@ class Node:
                     self.register_files()
                     self.last_file_refresh = time.time()
 
+            if self.settings_dirty and now > self.last_settings_flush + Node.SETTINGS_FLUSH_INTERVAL:
+                self.settings_dirty = False
+                self.last_settings_flush = now
+                try: self.app.save_peer_settings()
+                except Exception as e: RNS.log("Could not save peer settings: "+str(e), RNS.LOG_ERROR)
+
             time.sleep(self.job_interval)
 
     def peer_connected(self, link):
         RNS.log("Peer connected to "+str(self.destination), RNS.LOG_VERBOSE)
         try:
             self.app.peer_settings["node_connects"] += 1
-            self.app.save_peer_settings()
+            self.settings_dirty = True
 
         except Exception as e:
             RNS.log("Could not increase node connection count", RNS.LOG_ERROR)

nomadnet/NomadNetworkApp.py

@@ -8,6 +8,7 @@ import threading
 import traceback
 import subprocess
 import contextlib
+import tempfile
 
 import RNS
 import LXMF
@@ -41,6 +42,13 @@ class NomadNetworkApp:
         RNS.log("Saving directory...", RNS.LOG_VERBOSE)
         self.directory.save_to_disk()
 
+        if hasattr(self, "rrc") and self.rrc is not None:
+            try:
+                self.rrc.save()
+                self.rrc.shutdown()
+            except Exception:
+                pass
+
         if hasattr(self.ui, "restore_ixon"):
             if self.ui.restore_ixon:
                 try:
@@ -79,6 +87,8 @@ class NomadNetworkApp:
 
         self.uimode        = None
 
+        self.announce_interval = 6*60*60
+
         if configdir == None:
             self.configdir = NomadNetworkApp.configdir
         else:
@@ -102,22 +112,27 @@ class NomadNetworkApp:
         self.storagepath       = self.configdir+"/storage"
         self.identitypath      = self.configdir+"/storage/identity"
         self.cachepath         = self.configdir+"/storage/cache"
+        self.imagecachepath    = self.configdir+"/storage/cache/images/browser"
+        self.convcachepath     = self.configdir+"/storage/cache/images/converted_node"
+        self.dispcachepath     = self.configdir+"/storage/cache/images/converted_disp"
         self.resourcepath      = self.configdir+"/storage/resources"
         self.conversationpath  = self.configdir+"/storage/conversations"
         self.directorypath     = self.configdir+"/storage/directory"
         self.peersettingspath  = self.configdir+"/storage/peersettings"
         self.tmpfilespath      = self.configdir+"/storage/tmp"
+        self.attachmentpath    = self.configdir+"/storage/attachments"
 
         self.pagespath         = self.configdir+"/storage/pages"
         self.filespath         = self.configdir+"/storage/files"
-        self.cachepath         = self.configdir+"/storage/cache"
         self.examplespath      = self.configdir+"/examples"
 
         self.downloads_path    = os.path.expanduser("~/Downloads")
+        self.attachment_save_path = None
 
         self.firstrun               = False
         self.should_run_jobs        = True
         self.job_interval           = 5
+        self.peer_settings_lock     = threading.Lock()
         self.defer_jobs             = 90
         self.page_refresh_interval  = 0
         self.file_refresh_interval  = 0
@@ -127,6 +142,7 @@ class NomadNetworkApp:
         self.try_propagation_on_fail = True
         self.disable_propagation     = True
         self.notify_on_new_message   = True
+        self.compose_markdown        = True
 
         self.lxmf_max_propagation_size = None
         self.lxmf_max_sync_size        = None
@@ -141,6 +157,20 @@ class NomadNetworkApp:
         self.required_stamp_cost   = None
         self.accept_invalid_stamps = False
 
+        self.rrc_history_per_room_cap = 500
+        self.rrc_filter_loaded_history = True
+        self.rrc_max_accepted_resource_size = 256*1024
+        self.rrc_ephemeral_notices = 600
+        self.rrc_nick_colors = True
+        self.rrc_nick_colors_theme = None
+        self.rrc_mention_color = None
+        self.rrc_color_mention_timestamps = True
+        self.rrc_ui_justify_msgs = True
+        self.rrc_ui_space_msgs = False
+        self.rrc_ui_render_markdown = True
+        self.rrc_ui_render_micron = True
+        self.rrc_show_gutters = False
+        self.rrc_enable_esoterics = False
 
         if not os.path.isdir(self.storagepath):
             os.makedirs(self.storagepath)
@@ -148,6 +178,15 @@ class NomadNetworkApp:
         if not os.path.isdir(self.cachepath):
             os.makedirs(self.cachepath)
 
+        if not os.path.isdir(self.imagecachepath):
+            os.makedirs(self.imagecachepath)
+
+        if not os.path.isdir(self.convcachepath):
+            os.makedirs(self.convcachepath)
+
+        if not os.path.isdir(self.dispcachepath):
+            os.makedirs(self.dispcachepath)
+
         if not os.path.isdir(self.resourcepath):
             os.makedirs(self.resourcepath)
 
@@ -165,6 +204,9 @@ class NomadNetworkApp:
 
         if not os.path.isdir(self.tmpfilespath):
             os.makedirs(self.tmpfilespath)
+
+        if not os.path.isdir(self.attachmentpath):
+            os.makedirs(self.attachmentpath)
         else:
             self.clear_tmp_dir()
 
@@ -220,11 +262,13 @@ class NomadNetworkApp:
                 RNS.log("The contained exception was: %s" % (str(e)), RNS.LOG_ERROR)
                 nomadnet.panic()
 
-        if os.path.isfile(self.peersettingspath):
+        self.peer_settings = None
+        self.peer_settings_notice = None
+        for path in (self.peersettingspath, self.peersettingspath+".bak"):
+            if self.peer_settings != None or not os.path.isfile(path): continue
             try:
-                file = open(self.peersettingspath, "rb")
+                with open(path, "rb") as file:
-                self.peer_settings = msgpack.unpackb(file.read())
+                    self.peer_settings = msgpack.unpackb(file.read())
-                file.close()
 
                 if not "node_last_announce" in self.peer_settings:
                     self.peer_settings["node_last_announce"] = None
@@ -241,25 +285,38 @@ class NomadNetworkApp:
                 if not "served_page_requests" in self.peer_settings:
                     self.peer_settings["served_page_requests"] = 0
 
+                if not "served_media_requests" in self.peer_settings:
+                    self.peer_settings["served_media_requests"] = 0
+
                 if not "served_file_requests" in self.peer_settings:
                     self.peer_settings["served_file_requests"] = 0
 
+                if not "quit_confirm" in self.peer_settings:
+                    self.peer_settings["quit_confirm"] = True
+
+                self.peer_settings["announce_interval"] = self.announce_interval
+                if path != self.peersettingspath:
+                    self.peer_settings_notice = "Peer settings were restored from the last backup.\nCheck your display name and propagation node."
+
             except Exception as e:
-                RNS.logdest = RNS.LOG_STDOUT
+                RNS.log(f"Could not load local peer settings from {path}", RNS.LOG_ERROR)
-                RNS.log(f"Could not load local peer settings from {self.peersettingspath}", RNS.LOG_ERROR)
                 RNS.log(f"The contained exception was: {e}", RNS.LOG_ERROR)
-                RNS.log(f"This likely means that the peer settings file has become corrupt.", RNS.LOG_ERROR)
+                RNS.log(f"Moving the corrupt file to {path}.corrupt", RNS.LOG_ERROR)
-                RNS.log(f"You can try deleting the file at {self.peersettingspath} and restarting nomadnet.", RNS.LOG_ERROR)
+                try: os.replace(path, path+".corrupt")
-                nomadnet.panic()
+                except Exception: pass
-        else:
+                self.peer_settings = None
+                self.peer_settings_notice = "Peer settings were corrupt and have been reset.\nCheck your display name and propagation node."
+
+        if self.peer_settings == None:
             try:
-                RNS.log("No peer settings file found, creating new...")
+                RNS.log("Creating new peer settings file...")
                 self.peer_settings = {
                     "display_name": "Anonymous Peer",
-                    "announce_interval": None,
+                    "announce_interval": self.announce_interval,
                     "last_announce": None,
                     "node_last_announce": None,
                     "propagation_node": None,
+                    "quit_confirm": True,
                     "last_lxmf_sync": 0,
                     "node_connects": 0,
                     "served_page_requests": 0,
@@ -296,6 +353,13 @@ class NomadNetworkApp:
 
         self.directory = nomadnet.Directory(self)
 
+        from nomadnet.RRC import RRCManager
+        self.rrc = RRCManager(self)
+        self.rrc.load()
+
+        from nomadnet.InterfaceProfiles import InterfaceProfiles
+        self.interface_profiles = InterfaceProfiles(self)
+
         static_peers = []
         for static_peer in self.static_peers:
             try:
@@ -431,6 +495,9 @@ class NomadNetworkApp:
                 RNS.log("Initiating automatic LXMF sync", RNS.LOG_VERBOSE)
                 self.request_lxmf_sync(limit=self.lxmf_sync_limit)
 
+            if now > self.peer_settings["last_announce"] + self.peer_settings["announce_interval"]:
+                self.announce_now()
+
             time.sleep(self.job_interval)
 
     def set_display_name(self, display_name):
@@ -513,7 +580,54 @@ class NomadNetworkApp:
         if self.message_router.propagation_transfer_state != LXMF.LXMRouter.PR_IDLE:
             self.message_router.cancel_propagation_node_requests()
 
+    def _persist_ignored_list(self):
+        try:
+            with open(self.ignoredpath, "wb") as fh:
+                for h in self.ignored_list:
+                    fh.write((RNS.hexrep(h, delimit=False)+"\n").encode("utf-8"))
+        except Exception as e:
+            RNS.log("Could not persist ignored list: "+str(e), RNS.LOG_ERROR)
+
+    def block_destination(self, dest_hash, reason=None):
+        if not isinstance(dest_hash, (bytes, bytearray)):
+            return False
+        dest_hash = bytes(dest_hash)
+        try:
+            identity = RNS.Identity.recall(dest_hash)
+            if identity is not None:
+                RNS.Transport.blackhole_identity(identity.hash, reason=reason)
+        except Exception as e:
+            RNS.log("Could not blackhole identity: "+str(e), RNS.LOG_ERROR)
+        if dest_hash not in self.ignored_list:
+            self.ignored_list.append(dest_hash)
+            self._persist_ignored_list()
+        try:
+            self.message_router.ignore_destination(dest_hash)
+        except Exception:
+            pass
+        return True
+
+    def unblock_destination(self, dest_hash):
+        if not isinstance(dest_hash, (bytes, bytearray)):
+            return False
+        dest_hash = bytes(dest_hash)
+        try:
+            identity = RNS.Identity.recall(dest_hash)
+            if identity is not None:
+                RNS.Transport.unblackhole_identity(identity.hash)
+        except Exception as e:
+            RNS.log("Could not lift blackhole on identity: "+str(e), RNS.LOG_ERROR)
+        if dest_hash in self.ignored_list:
+            self.ignored_list.remove(dest_hash)
+            self._persist_ignored_list()
+        try:
+            self.message_router.unignore_destination(dest_hash)
+        except Exception:
+            pass
+        return True
+
     def announce_now(self):
+        RNS.log("Sending lxmf.delivery announce", RNS.LOG_VERBOSE)
         self.message_router.set_inbound_stamp_cost(self.lxmf_destination.hash, self.required_stamp_cost)
         self.lxmf_destination.display_name = self.peer_settings["display_name"]
         self.message_router.announce(self.lxmf_destination.hash)
@@ -542,9 +656,10 @@ class NomadNetworkApp:
         if selected_node == None:
             RNS.log("Could not autoselect a propagation node! LXMF propagation will not be available until a trusted node announces on the network, or a propagation node is manually selected.", RNS.LOG_WARNING)
         else:
-            pn_name_str = ""
+            if self.message_router.get_outbound_propagation_node() != selected_node:
-            RNS.log("Selecting "+RNS.prettyhexrep(selected_node)+pn_name_str+" as default LXMF propagation node", RNS.LOG_DEBUG)
+                pn_name_str = ""
-            self.message_router.set_outbound_propagation_node(selected_node)
+                RNS.log("Selecting "+RNS.prettyhexrep(selected_node)+pn_name_str+" as default LXMF propagation node", RNS.LOG_DEBUG)
+                self.message_router.set_outbound_propagation_node(selected_node)
 
     def get_user_selected_propagation_node(self):
         if "propagation_node" in self.peer_settings:
@@ -561,9 +676,16 @@ class NomadNetworkApp:
         return self.message_router.get_outbound_propagation_node()
 
     def save_peer_settings(self):
-        tmp_path = f"{self.peersettingspath}.tmp"
+        with self.peer_settings_lock:
-        with open(tmp_path, "wb") as file: file.write(msgpack.packb(self.peer_settings))
+            fd, tmp_path = tempfile.mkstemp(dir=os.path.dirname(self.peersettingspath), prefix=".peersettings.")
-        os.replace(tmp_path, self.peersettingspath)
+            try:
+                with os.fdopen(fd, "wb") as file: file.write(msgpack.packb(self.peer_settings))
+                if os.path.isfile(self.peersettingspath): os.replace(self.peersettingspath, self.peersettingspath+".bak")
+                os.replace(tmp_path, self.peersettingspath)
+            except Exception:
+                try: os.unlink(tmp_path)
+                except Exception: pass
+                raise
 
     def lxmf_delivery(self, message):
         time_string = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(message.timestamp))
@@ -673,16 +795,24 @@ class NomadNetworkApp:
             return False
 
     def conversation_is_unread(self, source_hash):
-        if bytes.fromhex(source_hash) in nomadnet.Conversation.unread_conversations:
+        src_bytes = bytes.fromhex(source_hash)
+        if src_bytes in nomadnet.Conversation.unread_conversations:
             return True
-        else:
+        if src_bytes in nomadnet.Conversation.failed_conversations:
-            return False
+            return True
+        return False
 
     def mark_conversation_read(self, source_hash):
-        if bytes.fromhex(source_hash) in nomadnet.Conversation.unread_conversations:
+        src_bytes = bytes.fromhex(source_hash)
-            nomadnet.Conversation.unread_conversations.pop(bytes.fromhex(source_hash))
+        if src_bytes in nomadnet.Conversation.unread_conversations:
+            nomadnet.Conversation.unread_conversations.pop(src_bytes)
             if os.path.isfile(self.conversationpath + "/" + source_hash + "/unread"):
                 os.unlink(self.conversationpath + "/" + source_hash + "/unread")
+        if src_bytes in nomadnet.Conversation.failed_conversations:
+            nomadnet.Conversation.failed_conversations.pop(src_bytes)
+            if os.path.isfile(self.conversationpath + "/" + source_hash + "/failed"):
+                try: os.unlink(self.conversationpath + "/" + source_hash + "/failed")
+                except Exception: pass
 
     def notify_message_recieved(self):
         if self.uimode == nomadnet.ui.UI_TEXT:
@@ -736,10 +866,19 @@ class NomadNetworkApp:
                     value = self.config["client"]["downloads_path"]
                     self.downloads_path = os.path.expanduser(value)
 
+                if option == "attachment_save_path":
+                    value = self.config["client"]["attachment_save_path"]
+                    self.attachment_save_path = os.path.expanduser(value)
+
                 if option == "announce_at_start":
                     value = self.config["client"].as_bool(option)
                     self.peer_announce_at_start = value
 
+                if option == "announce_interval":
+                    value = self.config["client"].as_int(option)
+                    if value < 30: value = 30
+                    self.announce_interval = value*60
+
                 if option == "try_propagation_on_send_fail":
                     value = self.config["client"].as_bool(option)
                     self.try_propagation_on_fail = value
@@ -796,6 +935,10 @@ class NomadNetworkApp:
                     value = self.config["client"].as_bool(option)
                     self.notify_on_new_message = value
 
+                if option == "compose_in_markdown":
+                    value = self.config["client"].as_bool(option)
+                    self.compose_markdown = value
+
                 if option == "user_interface":
                     value = value.lower()
                     if value == "none":
@@ -829,6 +972,16 @@ class NomadNetworkApp:
                             else:
                                 self.config["textui"]["hide_guide"] = self.config["textui"].as_bool("hide_guide")
 
+                            if not "sanitize_names" in self.config["textui"]:
+                                self.config["textui"]["sanitize_names"] = True
+                            else:
+                                self.config["textui"]["sanitize_names"] = self.config["textui"].as_bool("sanitize_names")
+
+                            if not "clipboard_copy" in self.config["textui"]:
+                                self.config["textui"]["clipboard_copy"] = False
+                            else:
+                                self.config["textui"]["clipboard_copy"] = self.config["textui"].as_bool("clipboard_copy")
+
                             if not "animation_interval" in self.config["textui"]:
                                 self.config["textui"]["animation_interval"] = 1
                             else:
@@ -850,6 +1003,20 @@ class NomadNetworkApp:
                                 else:
                                     raise ValueError("The selected Text UI color mode is invalid")
 
+                            if not "image_loading" in self.config["textui"]:
+                                self.config["textui"]["image_loading"] = nomadnet.ui.TextUI.IMGLOAD_AUTO
+                            else:
+                                if self.config["textui"]["image_loading"].lower() == "never":
+                                    self.config["textui"]["image_loading"] = nomadnet.ui.TextUI.IMGLOAD_NEVER
+                                elif self.config["textui"]["image_loading"].lower() == "manual":
+                                    self.config["textui"]["image_loading"] = nomadnet.ui.TextUI.IMGLOAD_MANUAL
+                                elif self.config["textui"]["image_loading"].lower() == "auto":
+                                    self.config["textui"]["image_loading"] = nomadnet.ui.TextUI.IMGLOAD_AUTO
+                                elif self.config["textui"]["image_loading"].lower() == "always":
+                                    self.config["textui"]["image_loading"] = nomadnet.ui.TextUI.IMGLOAD_ALWAYS
+                                else:
+                                    raise ValueError("The selected image loading mode is invalid")
+
                             if not "theme" in self.config["textui"]:
                                 self.config["textui"]["theme"] = nomadnet.ui.TextUI.THEME_DARK
                             else:
@@ -866,6 +1033,98 @@ class NomadNetworkApp:
                     if value == "web":
                         self.uimode = nomadnet.ui.UI_WEB
 
+        if not "utilities" in self.config:
+            self.config["utilities"] = {}
+        if not "enable_utilities" in self.config["utilities"]:
+            self.config["utilities"]["enable_utilities"] = False
+        else:
+            self.config["utilities"]["enable_utilities"] = self.config["utilities"].as_bool("enable_utilities")
+
+        if "rrc" in self.config:
+            for option in self.config["rrc"]:
+                if option == "history_per_room_cap":
+                    try: value = self.config["rrc"].as_int(option)
+                    except Exception: value = None
+                    if value is not None and value >= 0:
+                        self.rrc_history_per_room_cap = value
+
+                if option == "filter_loaded_history":
+                    try: value = self.config["rrc"].as_bool(option)
+                    except Exception: value = True
+                    self.rrc_filter_loaded_history = value
+
+                if option == "max_accepted_resource_size":
+                    try: value = self.config["rrc"].as_int(option)
+                    except Exception: value = None
+                    if value is not None and value >= 0:
+                        self.rrc_max_accepted_resource_size = value*1024
+
+                if option == "ephemeral_notices":
+                    try: value = self.config["rrc"].as_float(option)
+                    except Exception: value = 0
+                    self.rrc_ephemeral_notices = value*60
+
+                if option == "justify_msgs":
+                    try: value = self.config["rrc"].as_bool(option)
+                    except Exception: value = True
+                    self.rrc_ui_justify_msgs = value
+
+                if option == "space_msgs":
+                    try: value = self.config["rrc"].as_bool(option)
+                    except Exception: value = False
+                    self.rrc_ui_space_msgs = value
+
+                if option == "nick_colors":
+                    try: value = self.config["rrc"].as_bool(option)
+                    except Exception: value = True
+                    self.rrc_nick_colors = value
+
+                if option == "mention_color":
+                    value = self.config["rrc"][option]
+                    if len(value) == 6:
+                        try: bytes.fromhex(value)
+                        except Exception: value = None
+                        self.rrc_mention_color = value
+
+                if option == "color_mention_timestamps":
+                    try: value = self.config["rrc"].as_bool(option)
+                    except Exception: value = True
+                    self.rrc_color_mention_timestamps = value
+
+                if option == "nick_colors_theme":
+                    try:
+                        colors = self.config["rrc"].as_list(option)
+                        nick_colors_theme = []
+                        for c in colors:
+                            if len(c) == 6:
+                                try: bytes.fromhex(c)
+                                except: continue
+                                nick_colors_theme.append(c)
+                        self.rrc_nick_colors_theme = nick_colors_theme
+                    except Exception as e:
+                        RNS.log(f"Could not load custom nick colors theme: {e}", RNS.LOG_WARNING)
+                        self.rrc_nick_colors_theme = None
+
+                if option == "render_markdown":
+                    try: value = self.config["rrc"].as_bool(option)
+                    except Exception: value = True
+                    self.rrc_ui_render_markdown = value
+
+                if option == "render_micron":
+                    try: value = self.config["rrc"].as_bool(option)
+                    except Exception: value = True
+                    self.rrc_ui_render_micron = value
+
+                if option == "show_gutters":
+                    try: value = self.config["rrc"].as_bool(option)
+                    except Exception: value = False
+                    self.rrc_show_gutters = value
+
+                if option == "enable_esoterics":
+                    try: value = self.config["rrc"].as_bool(option)
+                    except Exception: value = False
+                    self.rrc_enable_esoterics = value
+
         if "node" in self.config:
             if not "enable_node" in self.config["node"]:
                 self.enable_node = False
@@ -1059,12 +1318,20 @@ destination = file
 enable_client = yes
 user_interface = text
 downloads_path = ~/Downloads
+
+# Where to save received attachments. If not set,
+# attachments will be saved to the downloads path.
+# attachment_save_path = ~/Downloads
 notify_on_new_message = yes
 
 # By default, the peer is announced at startup
 # to let other peers reach it immediately.
 announce_at_start = yes
 
+# Automatic announce interval in minutes for
+# your LXMF address. 6 hours by default.
+announce_interval = 360
+
 # By default, the client will try to deliver a
 # message via the LXMF propagation network, if
 # a direct delivery to the recipient is not
@@ -1118,6 +1385,10 @@ max_accepted_size = 500
 # been received, for every destination.
 compact_announce_stream = yes
 
+# You can choose whether or not to compose
+# messages in markdown format.
+compose_in_markdown = yes
+
 [textui]
 
 # Amount of time to show intro screen
@@ -1139,8 +1410,8 @@ theme = dark
 # colormode = monochrome
 # colormode = 16
 # colormode = 88
-colormode = 256
+# colormode = 256
-# colormode = 24bit
+colormode = 24bit
 
 # By default, unicode glyphs are used. If
 # you have a Nerd Font installed, you can
@@ -1149,8 +1420,17 @@ colormode = 256
 # your terminal doesn't support unicode.
 
 # glyphs = plain
-glyphs = unicode
+# glyphs = unicode
-# glyphs = nerdfont
+glyphs = nerdfont
+
+# If your terminal supports image output,
+# the page browser can display images and
+# graphics on pages. You can specify how,
+# or if at all, images are loaded. Valid
+# options are "auto", "manual", "never"
+# and "always".
+
+image_loading = auto
 
 # You can specify whether mouse events
 # should be considered as input to the
@@ -1164,6 +1444,74 @@ editor = nano
 # show up in the menu, you can disable it.
 hide_guide = no
 
+# You can select whether or not to perform
+# text sanitization on names received in
+# announces.
+sanitize_names = yes
+
+# You can enable clipboard features. This
+# relies on your terminal supporting OSC52
+# escape sequences.
+clipboard_copy = no
+
+[utilities]
+
+# You can enable a Utilities section in the
+# menu, where you can view the path and announce
+# rate tables and probe destinations. Disabled
+# by default.
+enable_utilities = no
+
+[rrc]
+
+# Maximum number of messages retained per
+# room in the in-memory scrollback buffer,
+# and the number of messages restored from
+# on-disk history at startup. The on-disk
+# log itself is appended to indefinitely;
+# this cap only controls how much backlog
+# is visible. Set to 0 to keep every message
+# in memory (full history).
+history_per_room_cap = 500
+
+# You can choose whether to filter system
+# and notice events when room history is
+# loaded.
+filter_loaded_history = yes
+
+# Maximum size in kilobytes of a resource
+# transfer that will be accepted from a
+# hub. Lower this on low-bandwidth links
+# to automatically reject transfers that
+# would take too long to receive. Set to
+# 0 to reject all resource transfers.
+max_accepted_resource_size = 256
+
+# You can choose whether notices and sys
+# messages persist indefinitely, or are
+# removed from the message history after
+# the specified time in minutes. Set to 0
+# to disable and keep forever.
+ephemeral_notices = 10
+
+# Other display and formatting options:
+color_mention_timestamps = yes
+render_markdown = yes
+render_micron = yes
+nick_colors = yes
+justify_msgs = yes
+space_msgs = no
+show_gutters = yes
+
+# You can configure your own color theme
+# for nick color assignment
+# nick_colors_theme = f68787, 00c394, d59e00, ...
+
+# You can set a specific color for mentions
+# of your nick
+# mention_color = FFBB44
+
+
 [node]
 
 # Whether to enable node hosting

nomadnet/_version.py

@@ -1 +1 @@
-__version__ = "0.9.2"
+__version__ = "1.4.4"

nomadnet/examples/various/input_fields.py

@@ -25,6 +25,8 @@ An masked field   : `B444`<!|password`Value of Field>`b
 
 An small field    : `B444`<8|small`test>`b, and some more text.
 
+A multi-row field : `B444`<40x4|notes`>`b
+
 Two fields        : `B444`<8|one`One>`b `B444`<8|two`Two>`b
 
 The data can be `!`[submitted`:/page/input_fields.mu`username|two]`!.

nomadnet/ui/TextUI.py

@@ -5,7 +5,9 @@ import os
 import platform
 
 import nomadnet
+from nomadnet.ui import THEME_DARK, THEME_LIGHT
 from nomadnet.ui.textui import *
+from nomadnet.ui.textui.images import ImageScreen
 from nomadnet import NomadNetworkApp
 
 COLORMODE_MONO = 1
@@ -13,8 +15,11 @@ COLORMODE_16   = 16
 COLORMODE_88   = 88
 COLORMODE_256  = 256
 COLORMODE_TRUE = 2**24
-THEME_DARK     = 0x01
+
-THEME_LIGHT    = 0x02
+IMGLOAD_NEVER  = 0x00
+IMGLOAD_MANUAL = 0x01
+IMGLOAD_AUTO   = 0x02
+IMGLOAD_ALWAYS = 0x03
 
 THEMES = {
     THEME_DARK: {
@@ -28,6 +33,7 @@ THEMES = {
             ("error_text",                  "dark red", "default",                  "default",                  "dark red", "default"),
             ("warning_text",                "yellow", "default",                    "default",                  "#ba4", "default"),
             ("inactive_text",               "dark gray", "default",                 "default",                  "dark gray", "default"),
+            ("browser_inactive",            "dark gray", "default",                 "default",                  "#444", "default"),
             ("buttons",                     "light green,bold", "default",          "default",                  "#00a533", "default"),
             ("msg_editor",                  "black", "light cyan",                  "standout",                 "#111", "#0bb"),
             ("msg_header_ok",               "black", "light green",                 "standout",                 "#111", "#6b2"),
@@ -37,6 +43,8 @@ THEMES = {
             ("msg_header_delivered",        "black", "light blue",                  "standout",                 "#111", "#28b"),
             ("msg_header_failed",           "black", "dark gray",                   "standout",                 "#000", "#777"),
             ("msg_warning_untrusted",       "black", "dark red",                    "standout",                 "#111", "dark red"),
+            ("msg_notice_unread",           "light blue", "default",                "standout",                 "#28b", "default"),
+            ("msg_notice_caution",          "yellow", "default",                    "standout",                 "#fd3", "default"),
             ("list_focus",                  "black", "light gray",                  "standout",                 "#111", "#aaa"),
             ("list_off_focus",              "black", "dark gray",                   "standout",                 "#111", "#777"),
             ("list_trusted",                "dark green", "default",                "default",                  "#6b2", "default"),
@@ -53,11 +61,20 @@ THEMES = {
             ("progress_empty",              "light gray", "default",                "default",                  "#ddd", "default"),
             ("interface_title",             "", "",                                 "default",                  "", ""),
             ("interface_title_selected",    "bold", "",                             "bold",                     "", ""),
+            ("interface_tile_focus",        "light blue,bold", "default",           "bold",                     "#5af,bold", "default"),
             ("connected_status",            "dark green", "default",                "default",                  "dark green", "default"),
-            ("disconnected_status",         "dark red", "default",                  "default",                  "dark red", "default"),
+            ("disconnected_status",         "dark red", "default",                   "default",                  "dark red", "default"),
+            ("disabled_status",             "dark red", "default",                   "default",                  "dark red", "default"),
             ("placeholder",                 "dark gray", "default",                 "default",                  "dark gray", "default"),
             ("placeholder_text",            "dark gray", "default",                 "default",                  "dark gray", "default"),
             ("error",                       "light red,blink",                      "default", "blink",         "#f44,blink", "default"),
+            ("irc_ts",                      "dark gray", "default",                 "default",                  "#888", "default"),
+            ("irc_nick_self",               "light green", "default",               "default",                  "#6c5", "default"),
+            ("irc_nick_peer",               "light cyan", "default",                "default",                  "#3cd", "default"),
+            ("irc_notice",                  "yellow", "default",                    "default",                  "#fd3", "default"),
+            ("irc_error",                   "light red", "default",                 "default",                  "#f55", "default"),
+            ("irc_system",                  "dark gray", "default",                 "default",                  "#888", "default"),
+            ("irc_mention",                 "light red,bold", "default",            "bold",                     "#fb4,bold", "default"),
 
         ],
     },
@@ -81,6 +98,8 @@ THEMES = {
             ("msg_header_delivered",        "black", "light blue",                 "standout",                 "#111", "#28b"),
             ("msg_header_failed",           "black", "dark gray",                  "standout",                 "#000", "#777"),
             ("msg_warning_untrusted",       "black", "dark red",                   "standout",                 "#111", "dark red"),
+            ("msg_notice_unread",           "dark blue", "default",               "standout",                  "#069", "default"),
+            ("msg_notice_caution",          "yellow", "default",                   "standout",                 "#fd3", "default"),
             ("list_focus",                  "black", "dark gray",                  "standout",                 "#111", "#aaa"),
             ("list_off_focus",              "black", "dark gray",                  "standout",                 "#111", "#777"),
             ("list_trusted",                "dark green", "default",               "default",                  "#4a0", "default"),
@@ -97,15 +116,41 @@ THEMES = {
             ("progress_empty",              "dark gray", "default",                "default",                  "#ddd", "default"),
             ("interface_title",             "dark gray", "default",                "default",                  "#444", "default"),
             ("interface_title_selected",    "dark gray,bold", "default",           "bold",                     "#444,bold", "default"),
+            ("interface_tile_focus",        "dark blue,bold", "default",           "bold",                     "#06c,bold", "default"),
             ("connected_status",            "dark green", "default",               "default",                  "#4a0", "default"),
-            ("disconnected_status",         "dark red", "default",                 "default",                  "#a22", "default"),
+            ("disconnected_status",         "dark red", "default",                  "default",                  "#a22", "default"),
+            ("disabled_status",             "dark red", "default",                  "default",                  "#a22", "default"),
             ("placeholder",                 "light gray", "default",               "default",                  "#999", "default"),
             ("placeholder_text",            "light gray", "default",               "default",                  "#999", "default"),
             ("error",                       "dark red,blink", "default",           "blink",                    "#a22,blink", "default"),
+            ("irc_ts",                      "dark gray", "default",                "default",                  "#888", "default"),
+            ("irc_nick_self",               "dark green", "default",               "default",                  "#3a0", "default"),
+            ("irc_nick_peer",               "dark cyan", "default",                "default",                  "#077", "default"),
+            ("irc_notice",                  "brown", "default",                    "default",                  "#a70", "default"),
+            ("irc_error",                   "dark red", "default",                 "default",                  "#a22", "default"),
+            ("irc_system",                  "dark gray", "default",                "default",                  "#888", "default"),
+            ("irc_mention",                 "dark red,bold", "default",             "bold",                     "#c50,bold", "default"),
         ],
     }
 }
 
+CONFIG_SECTION_COLORS = {
+    "logging":   ("yellow",        "#fd3", "brown",        "#a80"),
+    "client":    ("light cyan",    "#3cd", "dark cyan",    "#077"),
+    "textui":    ("light green",   "#6c5", "dark green",   "#2a0"),
+    "utilities": ("light gray",    "#bbb", "dark gray",    "#555"),
+    "rrc":       ("light magenta", "#d7f", "dark magenta", "#a2a"),
+    "node":      ("light blue",    "#5af", "dark blue",    "#06c"),
+    "printing":  ("brown",         "#b86", "brown",        "#852"),
+    "reticulum": ("light magenta", "#a6f", "dark magenta", "#63c"),
+}
+
+for _section, (_d16, _dhi, _l16, _lhi) in CONFIG_SECTION_COLORS.items():
+    THEMES[THEME_DARK]["urwid_theme"].append(("config_title_"+_section, _d16, "default", "default", _dhi, "default"))
+    THEMES[THEME_DARK]["urwid_theme"].append(("config_title_"+_section+"_selected", _d16+",bold", "default", "bold", _dhi+",bold", "default"))
+    THEMES[THEME_LIGHT]["urwid_theme"].append(("config_title_"+_section, _l16, "default", "default", _lhi, "default"))
+    THEMES[THEME_LIGHT]["urwid_theme"].append(("config_title_"+_section+"_selected", _l16+",bold", "default", "bold", _lhi+",bold", "default"))
+
 GLYPHSETS = {
     "plain": 1,
     "unicode": 2,
@@ -146,6 +191,20 @@ GLYPHS = {
     ("qrcode",          "QR",        "\u25a4",      "\uf029"),
     ("selected",        "[*] ",      "\u25CF",      "\u25CF"),
     ("unselected",      "[ ] ",      "\u25CB",      "\u25CB"),
+    ("focus_arrow",     ">",         "\u25B8",      "\u25B8"),
+    ("sep_dot",         "|",         "\u00B7",      "\u00B7"),
+    ("connected",       "+",         "\u2713",      "\U000f0201"),
+    ("disconnected",    "x",         "\u2717",      "\U000f0202"),
+    ("file",            "[F]",       "\u25a4",      "\uf15b"),
+    ("image",           "[I]",       "\u25a3",      "\uf1c5"),
+    ("audio",           "[~]",       "\u266b",      "\uf1c7"),
+    ("pin",             "*",         "\u2605",      "\uf08d"),
+    ("copy",            "[C]",       "\u29c9",      "\uf0c5"),
+    ("folder",          "[+]",       "\u25b8",      "\uf07b"),
+    ("folder_open",     "[-]",       "\u25be",      "\uf07c"),
+    ("fold_open",       "-",         "▾",      "▾"),
+    ("fold_closed",     "+",         "▸",      "▸"),
+    ("dropdown",        " [v]",      " ▾",          " "),
 }
 
 class TextUI:
@@ -192,7 +251,7 @@ class TextUI:
         for glyph in GLYPHS:
             self.glyphs[glyph[0]] = glyph[GLYPHSETS[glyphset]]
 
-        self.screen = urwid.raw_display.Screen()
+        self.screen = ImageScreen()
         self.screen.register_palette(self.palette)
         
         self.main_display = Main.MainDisplay(self, self.app)
@@ -226,6 +285,7 @@ class TextUI:
         self.set_colormode(colormode)
 
         self.main_display.start()
+        if intro_timeout <= 0 and self.app.peer_settings_notice: self.main_display.show_notice(self.app.peer_settings_notice)
         self.loop.run()
 
     def set_colormode(self, colormode):
@@ -237,10 +297,9 @@ class TextUI:
             self.restore_palette = True
 
     def unhandled_input(self, key):
-        if key == "ctrl q":
+        if key == "ctrl q":   self.main_display.quit()
-            raise urwid.ExitMainLoop
+        elif key == "ctrl e": pass
-        elif key == "ctrl e":
-            pass
 
     def display_main(self, loop, user_data):
         self.loop.widget = self.main_display.widget
+        if self.app.peer_settings_notice: self.main_display.show_notice(self.app.peer_settings_notice)

nomadnet/ui/__init__.py

@@ -8,6 +8,8 @@ pyc_modules = glob.glob(os.path.dirname(__file__)+"/*.pyc")
 modules     = py_modules+pyc_modules
 __all__ = list(set([os.path.basename(f).replace(".pyc", "").replace(".py", "") for f in modules if not (f.endswith("__init__.py") or f.endswith("__init__.pyc"))]))
 
+THEME_DARK     = 0x01
+THEME_LIGHT    = 0x02
 
 UI_NONE       = 0x00
 UI_MENU       = 0x01

nomadnet/ui/textui/FileBrowser.py

@@ -0,0 +1,234 @@
+import os
+import nomadnet
+import urwid
+
+class FileBrowserEntry(urwid.WidgetWrap):
+    signals = ["click"]
+
+    def __init__(self, name, full_path, is_dir=False, is_parent=False, selected=False):
+        self.full_path = full_path
+        self.name = name
+        self.is_dir = is_dir
+        self.is_parent = is_parent
+        self.selected = selected
+        g = nomadnet.NomadNetworkApp.get_shared_instance().ui.glyphs
+        if is_parent:
+            display = g["arrow_l"]+" .."
+        elif is_dir:
+            display = g["arrow_r"]+" "+name+"/"
+        elif selected:
+            display = g["check"]+" "+name
+        else:
+            display = "  "+name
+        self.text_widget = urwid.SelectableIcon(display, 0)
+        if is_dir or is_parent:
+            style = "list_trusted"
+            focus_style = "list_focus"
+        elif selected:
+            style = "list_trusted"
+            focus_style = "list_focus_trusted"
+        else:
+            style = "list_unknown"
+            focus_style = "list_focus"
+        display_widget = urwid.AttrMap(self.text_widget, style, focus_style)
+        super().__init__(display_widget)
+
+    def keypress(self, size, key):
+        if key == "enter":
+            self._emit("click")
+        else:
+            return key
+
+    def mouse_event(self, size, event, button, x, y, focus):
+        if button == 1 and urwid.util.is_mouse_press(event):
+            self._emit("click")
+            return True
+        return False
+
+class FileBrowser(urwid.WidgetWrap):
+    def __init__(self, on_done=None, on_cancel=None, on_change=None, title="Select File", multi_select=True, start_path=None, selected=None, max_file_size=None):
+        self.on_done_cb = on_done
+        self.on_cancel_cb = on_cancel
+        self.on_change_cb = on_change
+        self.multi_select = multi_select
+        self.max_file_size = max_file_size
+        self.selected = list(selected) if selected else []
+
+        app = nomadnet.NomadNetworkApp.get_shared_instance()
+        self.g = app.ui.glyphs
+        self.current_path = os.path.expanduser(start_path) if start_path else os.path.expanduser("~")
+
+        self.path_label = urwid.Text("")
+        self.status_label = urwid.Text("")
+        self.file_walker = urwid.SimpleFocusListWalker([])
+        self.file_listbox = urwid.ListBox(self.file_walker)
+
+        self.button_columns = urwid.Columns([
+            (urwid.WEIGHT, 0.45, urwid.Button("Done", on_press=self._dismiss)),
+            (urwid.WEIGHT, 0.1, urwid.Text("")),
+            (urwid.WEIGHT, 0.45, urwid.Button("Cancel", on_press=self._cancel)),
+        ])
+
+        header_pile = urwid.Pile([
+            self.path_label,
+            self.status_label,
+            urwid.Divider(self.g["divider1"]),
+        ])
+        footer_pile = urwid.Pile([
+            urwid.Divider(self.g["divider1"]),
+            self.button_columns,
+        ])
+
+        self._populate()
+
+        self.browser_frame = urwid.Frame(self.file_listbox, header=header_pile, footer=footer_pile)
+        self._main_widget = urwid.LineBox(self.browser_frame, title=title)
+        self._notice_open = False
+        super().__init__(self._main_widget)
+
+    def _show_notice(self, message, title="Notice"):
+        def dismiss(_b=None):
+            self._notice_open = False
+            self._w = self._main_widget
+
+        box = urwid.LineBox(urwid.Pile([
+            urwid.Text(message, align=urwid.CENTER),
+            urwid.Divider(),
+            urwid.Padding(urwid.Button("OK", on_press=dismiss), align=urwid.CENTER, width=8),
+        ]), title=title)
+        self._notice_open = True
+        self._w = urwid.Overlay(box, self._main_widget, align=urwid.CENTER, width=("relative", 75), valign=urwid.MIDDLE, height=urwid.PACK)
+
+    def _dismiss_notice(self):
+        self._notice_open = False
+        self._w = self._main_widget
+
+    def _update_status(self):
+        if self.selected:
+            names = [os.path.basename(p) for p in self.selected]
+            self.status_label.set_text("  "+self.g["file"]+" "+str(len(self.selected))+" selected: "+", ".join(names))
+        elif self.multi_select:
+            self.status_label.set_text("  No files selected")
+        else:
+            self.status_label.set_text("  Select a file")
+
+    def _populate(self):
+        self.path_label.set_text("  "+self.current_path)
+        self._update_status()
+
+        focus_pos = None
+        try:
+            focus_pos = self.file_listbox.focus_position
+        except Exception:
+            pass
+
+        entries = []
+        parent = os.path.dirname(self.current_path)
+        if parent != self.current_path:
+            entry = FileBrowserEntry("..", parent, is_parent=True)
+            urwid.connect_signal(entry, "click", self._entry_clicked, entry)
+            entries.append(entry)
+
+        try:
+            items = sorted(os.listdir(self.current_path))
+        except PermissionError:
+            entries.append(urwid.Text(("error_text", "  Permission denied")))
+            self.file_walker[:] = entries
+            return
+        except Exception:
+            entries.append(urwid.Text(("error_text", "  Cannot read directory")))
+            self.file_walker[:] = entries
+            return
+
+        dirs = []
+        files = []
+        for item in items:
+            if item.startswith("."):
+                continue
+            full = os.path.join(self.current_path, item)
+            if os.path.isdir(full):
+                dirs.append((item, full))
+            elif os.path.isfile(full):
+                files.append((item, full))
+
+        for name, full in dirs:
+            entry = FileBrowserEntry(name, full, is_dir=True)
+            urwid.connect_signal(entry, "click", self._entry_clicked, entry)
+            entries.append(entry)
+
+        for name, full in files:
+            entry = FileBrowserEntry(name, full, selected=(full in self.selected))
+            urwid.connect_signal(entry, "click", self._entry_clicked, entry)
+            entries.append(entry)
+
+        if not dirs and not files:
+            entries.append(urwid.Text(("inactive_text", "  (empty)")))
+
+        self.file_walker[:] = entries
+        if focus_pos is not None and focus_pos < len(entries):
+            self.file_listbox.set_focus(focus_pos)
+        elif entries:
+            self.file_listbox.set_focus(0)
+
+    def _entry_clicked(self, entry_widget, user_data=None):
+        entry = user_data if user_data else entry_widget
+        if entry.is_dir or entry.is_parent:
+            self.current_path = entry.full_path
+            self._populate()
+            return
+
+        if self.max_file_size is not None and entry.full_path not in self.selected:
+            try:
+                file_size = os.path.getsize(entry.full_path)
+            except Exception:
+                file_size = 0
+            if file_size > self.max_file_size:
+                limit_mb = self.max_file_size / (1024 * 1024)
+                size_mb = file_size / (1024 * 1024)
+                self._show_notice(
+                    "%s is %.1f MB.\nAttachments are limited to %.0f MB." % (entry.name, size_mb, limit_mb),
+                    title="File too large",
+                )
+                return
+
+        if self.multi_select:
+            if entry.full_path in self.selected:
+                self.selected.remove(entry.full_path)
+            else:
+                self.selected.append(entry.full_path)
+            if self.on_change_cb is not None:
+                self.on_change_cb(list(self.selected))
+            self._populate()
+        else:
+            self.selected = [entry.full_path]
+            if self.on_change_cb is not None:
+                self.on_change_cb(list(self.selected))
+            if self.on_done_cb is not None:
+                self.on_done_cb(list(self.selected))
+
+    def _dismiss(self, sender):
+        if self.on_done_cb is not None:
+            self.on_done_cb(list(self.selected))
+
+    def _cancel(self, sender):
+        self.selected = []
+        if self.on_cancel_cb is not None:
+            self.on_cancel_cb()
+
+    def keypress(self, size, key):
+        if self._notice_open:
+            if key in ("esc", "enter"):
+                self._dismiss_notice()
+                return None
+            return super().keypress(size, key)
+        if key == "esc":
+            self._cancel(None)
+            return None
+        result = super().keypress(size, key)
+        if result == "down" and self.browser_frame.focus_position == "body":
+            self.browser_frame.focus_position = "footer"
+            return None
+        elif result == "up" and self.browser_frame.focus_position == "footer":
+            self.browser_frame.focus_position = "body"
+            return None
+        return result

nomadnet/ui/textui/Guide.py

@@ -60,6 +60,75 @@ class SelectText(urwid.Text):
         self._emit('click')
         return True
 
+def _rows_above(attrmaps, index, cols):
+    if index <= 0 or not attrmaps:
+        return 0
+    total = 0
+    for i in range(min(index, len(attrmaps))):
+        try:
+            total += attrmaps[i].rows((cols,))
+        except Exception:
+            total += 1
+    return total
+
+
+class GuideColumns(urwid.Columns):
+    def keypress(self, size, key):
+        # TODO: Scroll handling is currently very weird
+        # in the guide reader. When using page up/down
+        # or mouse scroll, the actual cursor position
+        # does not follow the scroll, as it does in the
+        # browser. If you scroll down, and then use the
+        # arrow keys, the position jumps back up to
+        # where you "left" the cursor.
+        # There's handling for this in the Browser code
+        # but I don't have time to port it over here at
+        # this point. If someone else takes that one up
+        # it's :) :) :)
+        return super().keypress(size, key)
+
+
+class GuideLinkDelegate:
+    def __init__(self, app, reader=None):
+        self.app = app
+        self.g = app.ui.glyphs
+        self.reader = reader
+        self.last_keypress = 0
+
+    def marked_link(self, target, fields=None):
+        pass
+
+    def micron_released_focus(self):
+        self.reader.focus_topics()
+
+    def resolve_image(self, url):
+        if url.endswith("demo.webp"):
+            import os
+            import nomadnet.ui.textui.images as images_module
+            path = f"{os.path.dirname(images_module.__file__)}/demo.webp"
+            RNS.log(path)
+            return path
+
+        return None
+
+    def handle_link(self, target, fields=None):
+        if not target:
+            return
+        if target.startswith("#"):
+            if self.reader is not None:
+                try:
+                    def df(loop, user_data): self.reader.jump_to_anchor(target[1:])
+                    nomadnet.NomadNetworkApp.get_shared_instance().ui.loop.set_alarm_in(0.0, df)
+                except Exception as e:
+                    RNS.log("Guide anchor jump failed: "+str(e), RNS.LOG_ERROR)
+            return
+        try:
+            self.app.ui.main_display.show_network(None)
+            self.app.ui.main_display.sub_displays.network_display.browser.handle_link(target, fields)
+        except Exception as e:
+            RNS.log("Could not open guide link: "+str(e), RNS.LOG_ERROR)
+
+
 class GuideEntry(urwid.WidgetWrap):
     def __init__(self, app, parent, reader, topic_name):
         self.app = app
@@ -79,7 +148,7 @@ class GuideEntry(urwid.WidgetWrap):
 
     def display_topic(self, event, topic):
         markup = TOPICS[topic]
-        attrmaps = markup_to_attrmaps(markup, url_delegate=None)
+        attrmaps = markup_to_attrmaps(markup, url_delegate=GuideLinkDelegate(self.app, reader=self.reader))
 
         topic_position = None
         index = 0
@@ -109,11 +178,12 @@ class TopicList(urwid.WidgetWrap):
         self.topic_list = [
             GuideEntry(self.app, self, guide_display, "Introduction"),
             GuideEntry(self.app, self, guide_display, "Concepts & Terminology"),
+            GuideEntry(self.app, self, guide_display, "Channels & RRC"),
             GuideEntry(self.app, self, guide_display, "Interfaces"),
             GuideEntry(self.app, self, guide_display, "Hosting a Node"),
             GuideEntry(self.app, self, guide_display, "Configuration Options"),
             GuideEntry(self.app, self, guide_display, "Keyboard Shortcuts"),
-            GuideEntry(self.app, self, guide_display, "Markup"),
+            GuideEntry(self.app, self, guide_display, "Micron Markup"),
             self.first_run_entry,
             GuideEntry(self.app, self, guide_display, "Network Configuration"),
             GuideEntry(self.app, self, guide_display, "Display Test"),
@@ -148,13 +218,14 @@ class GuideDisplay():
         self.right_area = urwid.LineBox(urwid.Filler(topic_text, urwid.TOP))
 
 
-        self.columns = urwid.Columns(
+        self.columns = GuideColumns(
             [
                 (urwid.WEIGHT, GuideDisplay.list_width, self.left_area),
                 (urwid.WEIGHT, 1-GuideDisplay.list_width, self.right_area)
             ],
             dividechars=0, focus_column=0
         )
+        self.columns.guide_display = self
 
         self.shortcuts_display = GuideDisplayShortcuts(self.app)
         self.widget = self.columns
@@ -166,10 +237,53 @@ class GuideDisplay():
     def set_content_widgets(self, new_content):
         options = self.columns.options(width_type=urwid.WEIGHT, width_amount=1-GuideDisplay.list_width, box_widget=True)
         pile = urwid.Pile(new_content)
-        content = urwid.LineBox(urwid.AttrMap(ScrollBar(Scrollable(pile), thumb_char="\u2503", trough_char=" "), "scrollbar"))
+        self._content_attrmaps = new_content
+        self._content_scrollable = Scrollable(pile)
+        pile.automove_cursor_on_scroll = True
+        content = urwid.LineBox(urwid.AttrMap(ScrollBar(self._content_scrollable, thumb_char="\u2503", trough_char=" "), "scrollbar"))
 
         self.columns.contents[1] = (content, options)
 
+    def jump_to_anchor(self, name):
+        scrollable = getattr(self, "_content_scrollable", None)
+        attrmaps   = getattr(self, "_content_attrmaps",   None)
+        if scrollable is None or attrmaps is None:
+            return
+        anchors     = getattr(attrmaps, "anchors", None) or {}
+        header_rows = getattr(attrmaps, "header_rows", None) or []
+        cols = self._content_cols()
+        target_idx = None
+        if name:
+            target_idx = anchors.get(name)
+            if target_idx is None:
+                return
+        else:
+            try: current = scrollable.get_scrollpos()
+            except Exception: current = 0
+            for hr in header_rows:
+                if _rows_above(attrmaps, hr, cols) > current:
+                    target_idx = hr
+                    break
+            if target_idx is None:
+                return
+        row_offset = _rows_above(attrmaps, target_idx, cols)
+        try:
+            scrollable.force_cursor_update = True
+            scrollable.anchor_cursor_update = True
+            scrollable.set_scrollpos(int(row_offset))
+        except Exception:
+            pass
+
+    def _content_cols(self):
+        try: cols = self.app.ui.loop.screen.get_cols_rows()[0]
+        except Exception: cols = 100
+
+        try:
+            widths = self.columns.column_widths((cols,))
+            if len(widths) > 1 and widths[1] > 0: return max(40, widths[1] - 3)
+        except Exception: pass
+        return max(40, int(cols * (1 - GuideDisplay.list_width)) - 3)
+
     def shortcuts(self):
         return self.shortcuts_display
 
@@ -191,7 +305,7 @@ Nomad Network is build on LXMF and Reticulum, which together provides the crypto
 
 Nomad Network does not need any connections to the public internet to work. In fact, it doesn't even need an IP or Ethernet network. You can use it entirely over packet radio, LoRa or even serial lines. But if you wish, you can bridge islanded Reticulum networks over the Internet or private ethernet networks, or you can build networks running completely over the Internet. The choice is yours.
 
-The current version of the program should be considered a beta release. The program works well, but there will most probably be bugs and possibly sub-optimal performance in some scenarios. On the other hand, this is the best time to have an influence on the direction of the development of Nomad Network. To do so, join the discussion on the Nomad Network project on GitHub.
+The current version of the program should be considered a beta release. The program works well, but there will most probably be bugs and possibly sub-optimal performance in some scenarios. On the other hand, this is the best time to have an influence on the direction of the development of Nomad Network. To do so, join the discussion on the Nomad Network project at `_`F79d`[Aleph git`a8d24177d946de4f1f0a0fe1af9a1338:/page/index.mu]`f`_.
 
 '''
 
@@ -205,16 +319,68 @@ The different sections of the program has a number of keyboard shortcuts mapped,
  - Ctrl-E   Display and edit selected peer info
  - Ctrl-X   Delete conversation
  - Ctrl-R   Open LXMF syncronisation dialog
-
+ - Ctrl-U   Ingest LXMF URI
->>>Conversation Display
+ - Ctrl-O   Toggle sort mode
- - Ctrl-D   Send message
+ - Ctrl-P   Display own LXMF address
- - Ctrl-K   Clear input fields
+ - Ctrl-G   Toggle fullscreen
- - Ctrl-T   Toggle message title field
+
+>>>Message Editor
+ - Ctrl-D   Send message
+ - Ctrl-P   Compose paper message
+ - Ctrl-T   Toggle message title field
+ - Ctrl-F   Attach file
+ - Ctrl-S   Save focused attachment
+ - Tab      Switch focus to message list
+
+>>>Message List
+ - Ctrl-W   Close conversation
+ - Ctrl-U   Purge failed messages
  - Ctrl-O   Toggle sort mode
- - Ctrl-P   Purge failed messages
  - Ctrl-X   Clear conversation history
  - Ctrl-G   Toggle fullscreen conversation
- - Ctrl-W   Close conversation
+ - Ctrl-S   Save focused attachment
+ - Tab      Switch focus to message editor
+
+>>`!Channels Window`!
+>>>Channel List
+ - Ctrl-N   Add a new hub
+ - Ctrl-A   Add (join) a room
+ - Ctrl-R   Connect to selected hub
+ - Ctrl-W   Disconnect from selected hub
+ - Ctrl-T   Toggle auto-reconnect for selected hub
+ - Ctrl-E   Edit selected hub
+ - Ctrl-X   Remove selected hub
+ - F8       Toggle join/part collapse
+
+>>>Message Editor
+ - Ctrl-D   Send message
+ - Ctrl-X   Leave current room
+ - F8       Toggle join/part collapse
+ - Tab      Complete nickname
+
+>>>Message List
+ - Ctrl-X   Leave current room
+ - Ctrl-U   Toggle users pane
+ - Ctrl-Y   Toggle channel list
+ - F8       Toggle join/part collapse
+ - Tab      Switch focus to message editor
+
+>>`!Input Field Editing`!
+All text input fields support readline-style editing shortcuts. When an input
+field is focused, these take precedence over any window shortcut mapped to the
+same key (so, for example, Ctrl-U edits the line rather than toggling a pane):
+ - Ctrl-A      Move to beginning of line
+ - Ctrl-E      Move to end of line
+ - Ctrl-U      Delete from cursor to beginning of line
+ - Ctrl-K      Delete from cursor to end of line
+ - Ctrl-W      Delete previous word (whitespace-delimited)
+ - Ctrl-L      Delete the entire buffer
+ - Ctrl-Y      Paste (yank) most recently deleted text
+ - Ctrl-Left   Move backward one word
+ - Ctrl-Right  Move forward one word
+
+Text deleted with Ctrl-U, Ctrl-K, Ctrl-W or Ctrl-L is placed in a shared yank
+buffer that Ctrl-Y pastes back, so text can be moved between input fields.
 
 >>`!Network Window`!
 >>>Browser
@@ -343,9 +509,9 @@ By default, you can find the examples in `!~/.nomadnetwork/examples`!. If you bu
 
 >>Authenticating Users
 
-Sometimes, you don't want everyone to be able to view certain pages or execute certain scripts. In such cases, you can use `*authentication`* to control who gets to run certain requests.
+Sometimes, you don't want everyone to be able to view certain pages, download certain files or execute certain scripts. In such cases, you can use `*authentication`* to control who gets to run certain requests.
 
-To enable authentication for any page, simply add a new file to your pages directory with ".allowed" added to the file-name of the page. If your page is named "secret_page.mu", just add a file named "secret_page.mu.allowed".
+To enable authentication for any page or file, simply add a new file next to it with ".allowed" added to its file-name. If your page is named "secret_page.mu", just add a file named "secret_page.mu.allowed". The same works for files in your files directory, and the ".allowed" file itself is never served.
 
 For each user allowed to access the page, add a line to this file, containing the hash of that users primary identity. Users can find their own identity hash in the `![ Network ]`! part of the program, under `!Local Peer Info`!. If you want to allow access for three different users, your file would look like this:
 
@@ -416,8 +582,32 @@ To add a new interface, press `!< Ctrl + A >`!. From here you can select which t
 `Ffff`! (!) Note:`! After adding or modifying interfaces, you will need to restart NomadNet or your Reticulum instance for changes to take effect.`f`b
 
 
+>Interface Profiles
+
+Interface profiles let you save different combinations of enabled interfaces and switch between them quickly. For example, you might keep a `!Home`! profile that uses a local AutoInterface, and a `!Travel`! profile that reaches out over a TCP gateway instead.
+
+Press `!< Tab >`! from the `![ Interfaces ]`! section to move between the `!Interfaces`! and `!Profiles`! tabs.
+
+To save your current set of enabled interfaces as a new profile, press `!< Ctrl + S >`!. To assign an interface to one or more profiles, edit it with `!< Ctrl + E >`! and use the profile checkboxes in the edit form. You can also toggle the selected interface on or off directly with `!< Ctrl + O >`!, and filter the list by All, Enabled or Disabled with `!< Ctrl + F >`!.
+
+`Ffff`! (!) Note:`! Switching profiles changes which interfaces are enabled, so a restart of NomadNet or your Reticulum instance is required for the change to take effect.`f`b
+
+
 >Interface Types
 
+Jump to a specific interface type:
+
+ `F79d`_`[AutoInterface`#autointerface]`_`f
+ `F79d`_`[TCPClientInterface`#tcpclientinterface]`_`f
+ `F79d`_`[TCPServerInterface`#tcpserverinterface]`_`f
+ `F79d`_`[UDPInterface`#udpinterface]`_`f
+ `F79d`_`[I2PInterface`#i2pinterface]`_`f
+ `F79d`_`[RNodeInterface`#rnodeinterface]`_`f
+ `F79d`_`[RNodeMultiInterface`#rnodemultiinterface]`_`f
+ `F79d`_`[SerialInterface`#serialinterface]`_`f
+ `F79d`_`[KISSInterface`#kissinterface]`_`f
+ `F79d`_`[PipeInterface`#pipeinterface]`_`f
+
 >>AutoInterface
 
 The Auto Interface enables communication with other discoverable Reticulum nodes over autoconfigured IPv6 and UDP. It does not need any functional IP infrastructure like routers or DHCP servers, but will require at least some sort of switching medium between peers (a wired switch, a hub, a WiFi access point or similar).
@@ -449,7 +639,7 @@ I2P Tunneled: Enable for connecting through I2P
 KISS Framing: Enable for KISS framing for software modems
 ```
 
-This interface is commonly used to connect to Reticulum testnets or other persistent nodes on the Internet.
+This interface is commonly used to connect to Reticulum gateways or other persistent nodes on the Internet.
 
 >>TCPServerInterface
 
@@ -612,6 +802,61 @@ Conversations in Nomad Network
 '''
 
 
+TOPIC_CHANNELS = '''>Channels & RRC
+
+NomadNet includes a built-in client for `*RRC`* (Reticulum Relay Chat), a real-time text chat protocol that runs over Reticulum. The reference RRC server implementation lives at https://github.com/kc1awv/rrcd. Each RRC server is called a `!hub`!, and each hub hosts one or more `!rooms`! (channels) you can join. Hubs are addressed by a Reticulum destination hash, just like nodes and conversations.
+
+The Channels section of NomadNet (accessible from the main menu, or by pressing the corresponding shortcut) is where you manage your hubs, view connection status, and chat in rooms.
+
+>>Joining a hub
+
+To start chatting on a hub you must first add it to your hub list:
+
+>>>
+ - Open the `![ Channels ]`! section.
+ - Press the shortcut to open the `*New Hub`* dialog (shown in the shortcut bar at the bottom).
+ - Enter the `!hub address`! (the destination hash of the hub, typically 16 hex characters / 8 bytes) and an optional display name.
+ - Confirm the dialog. The hub will appear in your hub list.
+<
+
+You can sometimes receive a hub link from another user or from a node page. Activating such a link will pre-fill the New Hub dialog (and optionally a room name) for you.
+
+>>Connecting and listing rooms
+
+Once a hub is added:
+
+>>>
+ - Select the hub in the list and connect to it. Hubs can also be set to auto-reconnect.
+ - When connected, run `*/list`* to see public rooms hosted on this hub.
+ - Run `*/join <room>`* to join a room. Joined rooms appear under the hub in your channel list.
+<
+
+>>Chatting
+
+Inside a room, anything you type that does not start with a `*/`* is sent as a message to everyone in the room. The most common in-room commands are:
+
+>>>
+ - `*/help`*  show the full list of commands
+ - `*/who`*   list users in the current room
+ - `*/nick <name>`*   set your display name on this hub
+ - `*/topic <room> [text]`*   view or set the room topic
+ - `*/part [room]`*   leave a room (defaults to the current one)
+ - `*/quit`*   disconnect from the hub
+<
+
+Messages support both `*markdown`* and `*micron`* formatting, depending on your render settings (see the `*Configuration Options`* topic). Because the backtick is the micron control character, you must use the `!broken-bar`! character `*¦`* in place of `*\\``* when typing micron formatting in a message. The renderer converts `*¦`* back to `*\\``* for display.
+
+>>Mentions and privacy
+
+You will be notified (and the bell may ring, depending on your configuration) when another user mentions your nick with `*@yournick`*. RRC traffic is end-to-end encrypted by Reticulum between you and the hub, but other users in the same room can read what you write there. Treat rooms as semi-public spaces.
+
+>>Hub etiquette
+
+Each hub is operated independently and may have its own rules, MOTD, ban list and operator team. Be respecful, follow the hub's MOTD, and remember that operators can kick, ban and set modes on rooms they own.
+
+'''
+
+
 TOPIC_FIRST_RUN = '''>First Time Information
 
 Hi there. This first run message will only appear once. It contains a few pointers on getting started with Nomad Network, and getting the most out of the program.
@@ -622,7 +867,33 @@ To get the most out of Nomad Network, you will need a terminal that supports UTF
 
 It is recommended to use a terminal size of at least 135x32. Nomad Network will work with smaller terminal sizes, but the interface might feel a bit cramped.
 
-If you don't already have a Nerd Font installed (see https://www.nerdfonts.com/), I also highly recommend to do so, since it will greatly expand the amount of glyphs, icons and graphics that Nomad Network can use. Once you have your terminal set up with a Nerd Font, go to the `![ Config ]`! menu item and enable Nerd Fonts in the configuration instead of normal unicode glyphs.
+Nerd Fonts and true-color are enabled by default. To get the full visual experience you should have a Nerd Font installed in your terminal. You can verify this by visiting the `*Display Test`* topic in this guide and checking the `!Nerd Font Rendering Test`! section. If those glyphs render correctly, you are good to go.
+
+If they do not render, install a Nerd Font and configure your terminal to use it. A few common approaches:
+
+>>>
+ - `!Download from nerdfonts.com`!
+   Visit https://www.nerdfonts.com/font-downloads, pick a font you like
+   (Jet Brains Mono, Hack and Meslo are popular choices), unzip it
+   into `*~/.local/share/fonts`* (Linux) or install via Font Book (macOS),
+   then refresh the font cache with `*fc-cache -fv`* on Linux.
+
+ - `!Package manager`!
+   Many distros ship Nerd Font packages. On Arch: `*pacman -S ttf-nerd-fonts-symbols`*
+   or one of the per-family packages. On Debian/Ubuntu look for `*fonts-firacode`*
+   plus the symbols-only Nerd Font package, or install manually from the
+   nerdfonts.com release.
+
+ - `!Homebrew (macOS)`!
+   `*brew install --cask font-fira-code-nerd-font`* (or any other family).
+
+ - `!Configure your terminal`!
+   After installing, set your terminal emulator's font to the Nerd Font
+   variant (it usually has "Nerd Font" in the name). Restart NomadNet
+   so the new font are picked up.
+<
+
+If for some reason you cannot or do not want to use a Nerd Font, you can disable Nerd Font glyphs in the `![ Config ]`! menu and NomadNet will fall back to plain unicode symbols.
 
 Nomad Network expects that you are already connected to some form of Reticulum network. That could be as simple as the default one that Reticulum auto-generates on your local ethernet/WiFi network, or something much more complex. This short guide won't go into any details on building networks, but you will find other entries in the guide that deal with network setup and configuration.
 
@@ -717,6 +988,12 @@ Sets whether to output a notification character (bell or flash) to the terminal
 Determines whether your LXMF address is automatically announced when the program starts. Must be a boolean value.
 <
 
+>>>
+`!announce_interval = 360`!
+>>>>
+Determines how often, in minutes, your LXMF address is announced on the network. Defaults to 6 hours.
+<
+
 >>>
 `!try_propagation_on_send_fail = yes`!
 >>>>
@@ -765,6 +1042,72 @@ The maximum accepted unpacked size for messages received directly from other pee
 With this option enabled, Nomad Network will only display one entry in the announce stream per destination. Older announces are culled when a new one arrives.
 <
 
+>>>
+`!compose_in_markdown = yes`!
+>>>>
+Configures whether sent messages are composed in markdown format.
+<
+
+>> RRC Section
+
+This section hold configuration directives related to the RRC client behaviour. It is delimited by the `![rrc]`! header in the configuration file. Available directives, along with their default values, are as follows:
+
+>>>
+`!history_per_room_cap = 500`!
+>>>>
+Maximum number of messages retained per room in the in-memory scrollback buffer, and the number of messages restored from on-disk history at startup. The on-disk log itself is appended to indefinitely; this cap only controls how much backlog is visible. Set to 0 to keep every message in memory.
+<
+
+>>>
+`!filter_loaded_history = yes`!
+>>>>
+Whether to filter notices and system message when initially loading room history.
+<
+
+>>>
+`!ephemeral_notices = 10`!
+>>>>
+If enabled, notices and system messages will disappear from the history after a while. 0 disables, otherwise sets the timeout in minutes.
+<
+
+>>>
+`!render_markdown = yes`!
+>>>>
+Whether or not to render markdown formatting in messages.
+<
+
+>>>
+`!render_micron = yes`!
+>>>>
+Whether or not to render micron formatting in messages. When using micron in messages, use the ¦ character in place of backticks.
+<
+
+>>>
+`!nick_colors = yes`!
+>>>>
+Whether or not to render RRC nicks in distinct colors based on identity hash.
+<
+
+>>>
+`!mention_color = None`!
+>>>>
+Sets a specific color for mentions of your nick, for example FFBB44.
+<
+
+>>>
+`!color_mention_timestamps = yes`!
+>>>>
+Whether or not to color timestamps of messages you were mentioned in.
+<
+
+>>>
+`!justify_msgs = yes`!
+`!space_msgs = no`!
+`!show_gutters = no`!
+>>>>
+Rendering layout options for messages.
+<
+
 >> Text UI Section
 
 This section hold configuration directives related to the look and feel of the text-based user interface of the program. It is delimited by the `![textui]`! header in the configuration file. Available directives, along with their default values, are as follows:
@@ -803,6 +1146,18 @@ The `!nerdfont`! set allows using a much wider range of glyphs, icons and graphi
 Determines whether the program should react to mouse/touch input. Must be a boolean value.
 <
 
+>>>
+`!sanitize_names = yes`!
+>>>>
+This option allows sanitizing announced names before they are displayed in the user interface.
+<
+
+>>>
+`!clipboard_copy = no`!
+>>>>
+If your terminal supports it, you can enable copy-to-clipboard functionality using OSC52 escape sequences.
+<
+
 >>>
 `!hide_guide = no`!
 >>>>
@@ -827,6 +1182,17 @@ Tells the program what color palette is supported by the terminal. Most terminal
 `!24bit`! Most new terminals support this full-color mode
 <
 
+>>>
+`!image_loading = auto`!
+>>>>
+Specifies how, if at all, images on pages should be loaded. Available options:
+>>>>>
+`!never`! Never load any images
+`!manual`! Only load images when manually requseted with Ctrl-L
+`!auto`! Load images automatically if link conditions are sufficient
+`!always`! Always load images, regardless of link RTT and EDR
+<
+
 >>>
 `!theme = dark`!
 >>>>
@@ -976,24 +1342,11 @@ For future reference, you can download the Reticulum Manual in PDF format here:
 
 It might be nice to keep that handy when you are not connected to the Internet, as it is full of information and examples that are also very relevant to Nomad Network.
 
->The Reticulum Testnet
+>The Distributed Backbone
 
-If you have Internet access, and just want to get started experimenting, you are welcome to join the Unsigned.io RNS Testnet. The testnet is just that, an informal network for testing and experimenting. It will be up most of the time, and anyone can join, but it also means that there's no guarantees for service availability.
+If you have Internet access, and just want to get started experimenting, you can connected to the distributed global Reticulum backbone. For more information about this, see the Reticulum Manual, or sites such as https://directory.rns.recips or https://rmap.world.
 
-The Testnet also runs the latest version of Reticulum, often even a short while before it is publicly released, which means strange behaviour might occur. If none of that scares you, add the following interface to your Reticulum configuration file to join:
+If you connect to the public backbone, you can leave nomadnet running for a while and wait for it to receive announces from other nodes on the network that host pages or services, or you can try connecting directly to some nodes listed here:
-
->>
-[[RNS Testnet Dublin]]
-  type = TCPClientInterface
-  enabled = yes
-  target_host = dublin.connect.reticulum.network
-  target_port = 4965
-<
-
-If you connect to the testnet, you can leave nomadnet running for a while and wait for it to receive announces from other nodes on the network that host pages or services, or you can try connecting directly to some nodes listed here:
-
- - Dublin Hub Testnet Node    : `!`[abb3ebcd03cb2388a838e70c001291f9]`!
- - Frankfurt Hub Testnet Node : `!`[ea6a715f814bdc37e56f80c34da6ad51]`!
 
 To browse pages on a node that is not currently known, open the URL dialog in the `![ Network ]`! section of the program by pressing `!Ctrl+U`!, paste or enter the address and select `!< Go >`! or press enter. Nomadnet will attempt to discover and connect to the requested node. You can save the currently connected node by pressing `!Ctrl+S`!.
 '''
@@ -1033,6 +1386,32 @@ The following line should contain a grayscale gradient bar:
 Unicode Glyphs   : \u2713  \u2715  \u26a0  \u24c3  \u2193
 
 Nerd Font Glyphs : \uf484  \U000f04c5  \U000f0219  \U000f0002  \uf415  \uf023  \uf06e
+
+
+>>Nerd Font Rendering Test
+
+`cSince Nerd Font glyphs and true-color are enabled by default, the rows below should render as crisp icons rather than empty squares, question marks, or fallback ASCII. If any glyph appears blank or boxed, your terminal is not using a Nerd Font, or the font is missing the required glyph range.
+``
+
+>>>
+Common UI icons   : \uf484  \uf415  \uf023  \uf06e  \uf055  \uf056  \uf059
+Status / state    : \U000f04c5  \U000f0219  \U000f0002  \U000f1397  \U000f12fc  \U000f0156
+Network / radio   : \U000f05a9  \U000f05aa  \U000f05ab  \U000f05ac  \uf1eb  \uf012
+People / messaging: \uf0c0  \uf007  \uf2bd  \uf0e0  \uf27a  \uf086
+Devices / files   : \uf233  \uf07b  \uf15b  \uf019  \uf093  \uf1c0
+<
+
+`cIf the above renders correctly, you have a working Nerd Font setup and can leave the defaults as-is. If not, see the `*First Time Information`* topic for install instructions.
+``
+
+>>Image Rendering Test
+
+`(The RNS logo`w=n`a=c`:/media/demo.webp)
+
+If you see the RNS logo above, your terminal supports image rendering! If not, you can use a terminal that supports the Kitty Terminal Graphics Protocol, for example Kitty, Konsole, Wezterm, iTerm2 and others.
+
+It is also important to note, that for image rendering to work, your operating system will need `*some sort`* of mechanism to actually decode the WebP format. Nomadnet will attempt to detect a suitable decoding backend on your system, and should be able to do so on 99% of variants across Linux, BSD, macOS and Windows, as long as `Faaalibwebp`f or one of the fallback decoders are available. If all else fails, you can install PIL (`Faaapip install pillow`f) or download the open-source `Faaadwebp`f decoder utility and place it on your path (for example at `Faaa~/.local/bin/dwebp`f); nomadnet will auto-detect either.
+
 '''
 
 
@@ -1091,6 +1470,24 @@ Nomad Network supports a simple and functional markup language called `*micron`*
 
 With micron you can easily create structured documents and pages with formatting, colors, glyphs and icons, ideal for display in terminals.
 
+>Table of Contents
+
+ `F79d`_`[A Few Demo Outputs`#a-few-demo-outputs]`_`f
+ `F79d`_`[Micron Tags`#micron-tags]`_`f
+ `F79d`_`[High Level Stuff`#high-level-stuff]`_`f
+ `F79d`_`[Collapsible Sections`#collapsible-sections]`_`f
+ `F79d`_`[Colors`#colors]`_`f
+ `F79d`_`[Page Foreground and Background Colors`#page-foreground-and-background-colors]`_`f
+ `F79d`_`[Links`#links]`_`f
+ `F79d`_`[Anchors`#anchors]`_`f
+ `F79d`_`[Tables`#tables]`_`f
+ `F79d`_`[Images`#images]`_`f
+ `F79d`_`[Fields & Requests`#fields-requests]`_`f
+ `F79d`_`[Comments`#comments]`_`f
+ `F79d`_`[Partials`#partials]`_`f
+ `F79d`_`[Literals`#literals]`_`f
+ `F79d`_`[Closing Remarks`#closing-remarks]`_`f
+
 >>Recommendations and Requirements
 
 While micron can output formatted text to even the most basic terminal, there's a few capabilities your terminal `*must`* support to display micron output correctly, and some that, while not strictly necessary, make the experience a lot better.
@@ -1247,6 +1644,32 @@ A sub sub section. We could continue, but you get the point.
 Wait! It's worth noting that we can also create sections without headings. They look like this.
 ``
 
+>Collapsible Sections
+
+Any section heading can be made collapsible, turning it into an accordion the reader can fold away and reveal on demand. Prefix a heading with \\`+ to make it foldable and open by default, or with \\`- to make it foldable but start collapsed:
+
+`Faaa
+`=
+`+>Open by default
+This text can be folded away by selecting the heading.
+
+`->Hidden until expanded
+This starts collapsed, and appears when the reader expands it.
+`=
+``
+
+A collapsible heading folds everything beneath it, up to the next heading of the same or a higher level, exactly like a normal section. It can therefore contain anything: text, tables, input fields, sub-sections, and even live partials.
+
+To fold or unfold a section, move onto its heading and press Enter or Space, or click it with the mouse. Open sections are marked with ▾ and collapsed ones with ▸.
+
+You can choose your own indicators for a page by placing a #!fold directive near the top, giving the open glyph followed by the collapsed glyph:
+
+`Faaa
+`=
+#!fold [-] [+]
+`=
+``
+
 
 >Colors
 
@@ -1284,7 +1707,7 @@ Here is a link without any label: `[72914442a3689add83a09a767963f57c:/page/index
 
 This is a `[labeled link`72914442a3689add83a09a767963f57c:/page/index.mu] to the same page, but it's hard to see if you don't know it
 
-Here is `F00a`_`[a more visible link`72914442a3689add83a09a767963f57c:/page/index.mu]`_`f
+Here is `F79d`_`[a more visible link`72914442a3689add83a09a767963f57c:/page/index.mu]`_`f
 `=
 ``
 
@@ -1296,15 +1719,124 @@ Here is a link without any label: `[72914442a3689add83a09a767963f57c:/page/index
 
 This is a `[labeled link`72914442a3689add83a09a767963f57c:/page/index.mu] to the same page, but it's hard to see if you don't know it
 
-Here is `F00f`_`[a more visible link`72914442a3689add83a09a767963f57c:/page/index.mu]`_`f
+Here is `F79d`_`[a more visible link`72914442a3689add83a09a767963f57c:/page/index.mu]`_`f
 
 ``
 
 When links like these are displayed in the built-in browser, clicking on them or activating them using the keyboard will cause the browser to load the specified URL.
 
+>Anchors
+
+Anchors let you create jump points within a single page similar to anchors in HTML. You declare a position in the page with a name, then link to it from anywhere on the same page.
+
+>>Auto-anchors from headers
+
+Every section heading you write also becomes an anchor automatically. The anchor name is the heading text after `*slugifying`*: lowercased, with any run of non-alphanumeric characters replaced by a single hyphen, and leading or trailing hyphens stripped. So `*>Hello World`* becomes the anchor `*hello-world`*, and `*>Introduction & Setup`* becomes `*introduction-setup`*.
+
+>>Explicit anchors
+
+If you want an anchor, that isn't tied to a heading, place one anywhere in your text with the \\`: tag, followed by a name. Names may contain the characters `*A-Z`*, `*a-z`*, `*0-9`*, `*_`* and `*-`*, and end at any other character (a space, a newline, or punctuation).
+
+The anchor itself takes up no space and does not render. Is's just a position marker. An explicit anchor declared on an otherwise empty line binds to that line's position.
+
+`Faaa
+`=
+`:install-notes
+Some installation notes for the user.
+
+You can also drop one mid-line. Example: see `:tip-3 ⚠ tip 3 below for caveats.
+`=
+``
+
+>>Linking to an anchor
+
+Reuse the standard link syntax, with a `*#`*-prefixed URL:
+
+`Faaa
+`=
+`[Jump to Install Notes`#install-notes]
+`=
+``
+
+When the user activates the link the browser scrolls the current page to the anchor's row.
+
+>>Jumping to the next section
+
+If the URL is just `*#`* with no name after it, the link jumps to the next \\`> header that appears after the link's own position in the document. This is convenient for "Continue ↓" buttons after a long paragraph, without having to name every section:
+
+`Faaa
+`=
+`[Continue`#]
+`=
+``
+
+>>Anchors in external links
+
+If you want to link to an anchor on another page, you can include it as a request variable:
+
+`Faaa
+`=
+`[Conclusion`a8d24177d946de4f1f0a0fe1af9a1338:/page/document.mu`anchor=conclusion]
+`=
+``
+
+>>Notes on namespaces and collisions
+
+Auto-anchors from headings and explicit \\`: anchors share a single namespace per page. If an explicit anchor collides with a heading slug, the first one declared is where it will jump to.
+
+>Tables
+
+You can include rendered tables by enclosing them in \\`t tags. Optionally, you can also specify alignment and max rendering width by adding these properties to the opening \\`t tag, like \\`tc30. Here's an example:
+
+`Faaa
+`=
+`t
+| Name | Price | Qty |
+| ---- | :---: | --: |
+| `F3a3Apple`f | Free | `!5`! |
+| Orange | Ask, nicely | 3 |
+`t
+`=
+``
+
+The above markup produces the following table:
+
+`t
+| Name | Price | Qty |
+| ---- | :---: | --: |
+| `F3a3Apple`f | Free | `!5`! |
+| Orange | Ask, nicely | 3 |
+`t
+
+>Images
+
+You can include images in pages by using the image tag. All images must specify at least an alt-text and a URL, and can optionally specify rendering options such as width, height and alignment. The alt-text is required so a meaningful placeholder can be displayed for clients that do not support image rendering.
+
+Width (`Faaaw=`f) and height (`Faaah=`f) can be specified either in terminal columns/rows, in percentages, or using the `Faaan`f specifier, which will render the image as close to its native resolution as possible
+
+Alignment (`Faaaa=`f) can be either `Faaaleft`f (`Faaal`f), `Faaaright`f (`Faaar`f) or `Faaacenter`f (`Faaac`f).
+
+`Faaa
+`=
+`(The RNS logo`w=n`a=c`:/media/demo.webp)
+`=
+``
+
+Images are block-level elements, and must be placed independently on their own line.
+
+If your terminal supports image rendering, the image will be displayed as specified. If not, a text placeholder is shown instead:
+
+`(The RNS logo`w=n`a=c`:/media/demo.webp)
+
+If you see the image above, your terminal supports image rendering! If not, you can use a terminal that supports the Kitty Terminal Graphics Protocol, for example Kitty, Konsole, Wezterm, iTerm2 and others.
+
+It is also important to note, that for image rendering to work, your operating system will need `*some sort`* of mechanism to actually decode the WebP format. Nomadnet will attempt to detect a suitable decoding backend on your system, and should be able to do so on 99% of variants across Linux, BSD, macOS and Windows, as long as `Faaalibwebp`f or one of the fallback decoders are available. If all else fails, you can install PIL (`Faaapip install pillow`f) or download the open-source `Faaadwebp`f decoder utility and place it on your path (for example at `Faaa~/.local/bin/dwebp`f); nomadnet will auto-detect either.
+
+`!Important!`! For the sake of bandwidth efficiency, images sent over the network `!must`! be in WebP format. Nomadnet will reject and ignore all other image formats. You can easily convert images to WebP with command-line utilities such as `Faaaconvert`f and similar. The WebP format supports both efficient compression and lossless alpha masks, so it is suitable for anything you can achieve with PNG, JPEG or other formats, but much lighter on bandwidth.
+
 >Fields & Requests
 
-Nomad Network let's you use simple input fields for submitting data to node-side applications. Submitted data, along with other session variables will be available to the node-side script / program as environment variables.
+Nomad Network lets you use simple input fields for submitting data to node-side applications. Submitted data, along with other session variables will be available to the node-side script / program as environment variables.
 
 >>Request Links
 
@@ -1358,6 +1890,14 @@ A sized input field:  `B444`<16|with_size`>`b
 `=
 ``
 
+You can also reserve more than one row for a field by appending `!xROWS`! to the width, which is useful for longer, multi-line input. This creates a field that is 40 columns wide and 5 rows tall:
+
+`Faaa
+`=
+A multi-row input field: `B444`<40x5|notes`>`b
+`=
+``
+
 It is possible to mask fields, for example for use with passwords and similar:
 
 `Faaa
@@ -1386,6 +1926,8 @@ A sized input field:  `B444`<16|with_size`>`B333
 
 A masked input field: `B444`<!|masked_demo`hidden text>`B333
 
+A multi-row input field: `B444`<40x5|notes`>`B333
+
 Full control: `B444`<!32|all_options`hidden text>`B333
 `b
 
@@ -1455,6 +1997,36 @@ This line will
 ``
 
 
+>Partials
+
+You can include partials in pages, which will load asynchronously once the page itself has loaded.
+
+`Faaa
+`=
+`{f64a846313b874ee4a357040807f8c77:/page/partial_1.mu}
+`=
+``
+
+It's also possible to set an auto-refresh interval for partials. Omit or set to 0 to disable. The following partial will update every 10 seconds.
+
+`Faaa
+`=
+`{f64a846313b874ee4a357040807f8c77:/page/refreshing_partial.mu`10}
+`=
+``
+
+You can include field values and variables in partial updates, and by setting the `!pid`! variable, you can create links that update one or more specific partials.
+
+`Faaa
+`=
+Name: `B444`<user_name`>`b
+
+`F38a`[Say hello`p:32]`f
+
+`{f64a846313b874e84a357039807f8c77:/page/hello_partial.mu`0`pid=32|user_name}
+`=
+``
+
 >Literals
 
 To display literal content, for example source-code, or blocks of text that should not be interpreted by micron, you can use literal blocks, specified by the \\`= tag. Below is the source code of this entire document, presented as a literal block.
@@ -1470,12 +2042,13 @@ TOPIC_MARKUP += "\n`=\n\n>Closing Remarks\n\nIf you made it all the way here, yo
 TOPICS = {
     "Introduction": TOPIC_INTRODUCTION,
     "Concepts & Terminology": TOPIC_CONCEPTS,
+    "Channels & RRC": TOPIC_CHANNELS,
     "Conversations": TOPIC_CONVERSATIONS,
     "Interfaces": TOPIC_INTERFACES,
     "Hosting a Node": TOPIC_HOSTING,
     "Configuration Options": TOPIC_CONFIG,
     "Keyboard Shortcuts": TOPIC_SHORTCUTS,
-    "Markup": TOPIC_MARKUP,
+    "Micron Markup": TOPIC_MARKUP,
     "Display Test": TOPIC_DISPLAYTEST,
     "Network Configuration": TOPIC_NETWORKS,
     "Credits & Licenses": TOPIC_LICENSES,

nomadnet/ui/textui/Helpers.py

@@ -0,0 +1,46 @@
+import sys
+import base64
+import urwid
+import RNS
+
+
+def osc52_copy(text):
+    if not text:
+        return False
+    try:
+        encoded = base64.b64encode(text.encode("utf-8")).decode("ascii")
+        sys.stdout.write("\x1b]52;c;" + encoded + "\x07")
+        sys.stdout.flush()
+        return True
+    except Exception as e:
+        RNS.log("Could not emit clipboard escape sequence: "+str(e), RNS.LOG_ERROR)
+        return False
+
+
+def qr_ascii(data):
+    try:
+        import qrcode
+        import io
+        qr = qrcode.QRCode(error_correction=qrcode.constants.ERROR_CORRECT_L, box_size=1, border=1)
+        qr.add_data(data)
+        qr.make()
+        buf = io.StringIO()
+        qr.print_ascii(out=buf, invert=False)
+        return buf.getvalue().rstrip("\n")
+    except Exception as e:
+        RNS.log("QR generation failed: "+str(e), RNS.LOG_ERROR)
+        return None
+
+
+class ClickableIcon(urwid.Text):
+    _selectable = False
+
+    def __init__(self, text, on_click=None):
+        super().__init__(text)
+        self._on_click = on_click
+
+    def mouse_event(self, size, event, button, x, y, focus):
+        if button == 1 and "press" in event and self._on_click is not None:
+            self._on_click()
+            return True
+        return False

nomadnet/ui/textui/Log.py

@@ -4,6 +4,7 @@ import itertools
 import mmap
 import urwid
 import nomadnet
+from nomadnet.util import strip_modifiers
 
 
 class LogDisplayShortcuts():
@@ -55,6 +56,10 @@ class LogTerminal(urwid.WidgetWrap):
 
 
     def keypress(self, size, key):
+        if key == "ctrl q":
+            nomadnet.NomadNetworkApp.get_shared_instance().ui.main_display.quit()
+            return None
+
         if key == "up":
             nomadnet.NomadNetworkApp.get_shared_instance().ui.main_display.frame.focus_position = "header"
 
@@ -64,7 +69,7 @@ class LogTerminal(urwid.WidgetWrap):
 class LogTail(urwid.WidgetWrap):
     def __init__(self, app):
         self.app = app
-        self.log_tail = urwid.Text(tail(self.app.logfilepath, 50))
+        self.log_tail = urwid.Text(strip_modifiers(tail(self.app.logfilepath, 50)))
         self.log = urwid.Scrollable(self.log_tail)
         self.log.set_scrollpos(-1)
         self.log_scrollbar = urwid.ScrollBar(self.log)

nomadnet/ui/textui/Main.py

@@ -2,12 +2,16 @@ import RNS
 
 from .Network import *
 from .Conversations import *
+from .Channels import *
 from .Directory import *
 from .Config import *
 from .Interfaces import *
+from .Transport import *
+from .Utilities import *
 from .Map import *
 from .Log import *
 from .Guide import *
+from .Editor import *
 import urwid
 
 class SubDisplays():
@@ -15,18 +19,27 @@ class SubDisplays():
         self.app = app
         self.network_display = NetworkDisplay(self.app)
         self.conversations_display = ConversationsDisplay(self.app)
+        self.channels_display = ChannelsDisplay(self.app)
         self.directory_display = DirectoryDisplay(self.app)
         self.config_display = ConfigDisplay(self.app)
         self.interface_display = InterfaceDisplay(self.app)
+        self.utilities_display = UtilitiesDisplay(self.app) if self._utilities_enabled(app) else None
         self.map_display = MapDisplay(self.app)
         self.log_display = LogDisplay(self.app)
         self.guide_display = GuideDisplay(self.app)
+        self.page_editor_display = PageEditorDisplay(self.app) if app.enable_node else None
 
         if app.firstrun:
             self.active_display = self.guide_display
         else:
             self.active_display = self.conversations_display
 
+    def _utilities_enabled(self, app):
+        try:
+            return app.config["utilities"].as_bool("enable_utilities")
+        except Exception:
+            return False
+
     def active(self):
         return self.active_display
 
@@ -83,6 +96,18 @@ class MainFrame(urwid.Frame):
 
         return super(MainFrame, self).keypress(size, key)
 
+class QuitDialogLineBox(urwid.LineBox):
+    def __init__(self, *args, on_cancel=None, **kwargs):
+        self.on_cancel = on_cancel
+        super().__init__(*args, **kwargs)
+
+    def keypress(self, size, key):
+        if key == "esc":
+            if self.on_cancel is not None:
+                self.on_cancel()
+            return None
+        return super().keypress(size, key)
+
 class MainDisplay():
     def __init__(self, ui, app):
         self.ui = ui
@@ -103,6 +128,11 @@ class MainDisplay():
         self.sub_displays.active_display = self.sub_displays.conversations_display
         self.update_active_sub_display()
 
+    def show_channels(self, user_data):
+        self.sub_displays.active_display = self.sub_displays.channels_display
+        self.update_active_sub_display()
+        self.sub_displays.channels_display.start()
+
     def show_directory(self, user_data):
         self.sub_displays.active_display = self.sub_displays.directory_display
         self.update_active_sub_display()
@@ -120,6 +150,13 @@ class MainDisplay():
         self.update_active_sub_display()
         self.sub_displays.interface_display.start()
 
+    def show_utilities(self, user_data):
+        if self.sub_displays.utilities_display is None:
+            return
+        self.sub_displays.active_display = self.sub_displays.utilities_display
+        self.update_active_sub_display()
+        self.sub_displays.utilities_display.start()
+
     def show_log(self, user_data):
         self.sub_displays.active_display = self.sub_displays.log_display
         self.sub_displays.log_display.show()
@@ -129,6 +166,14 @@ class MainDisplay():
         self.sub_displays.active_display = self.sub_displays.guide_display
         self.update_active_sub_display()
 
+    def show_page_editor(self, user_data):
+        if not self.app.enable_node:
+            return
+        self.sub_displays.active_display = self.sub_displays.page_editor_display
+        self.update_active_sub_display()
+        self.sub_displays.page_editor_display.start()
+        self.frame.focus_position = "body"
+
     def update_active_sub_display(self):
         self.frame.contents["body"] = (self.sub_displays.active().widget, None)
         self.update_active_shortcuts()
@@ -149,6 +194,71 @@ class MainDisplay():
         self.menu_display.start()
 
     def quit(self, sender=None):
+        if self.app.peer_settings["quit_confirm"]: self.show_quit_dialog()
+        else: self.do_quit()
+
+    def show_quit_dialog(self, sender=None):
+        if getattr(self, "_quit_dialog_open", False): return
+
+        active = self.sub_displays.active_display
+        if active is self.sub_displays.log_display:
+            self.do_quit()
+            return
+        if active is self.sub_displays.config_display and self.sub_displays.config_display.editor_term is not None:
+            self.do_quit()
+            return
+
+        dont_ask_again = urwid.CheckBox("Don't ask again")
+
+        def confirm(button=None):
+            if dont_ask_again.get_state():
+                try:
+                    self.app.peer_settings["quit_confirm"] = False
+                    self.app.save_peer_settings()
+                except Exception: pass
+            self.do_quit()
+
+        def cancel(button=None):
+            self._quit_dialog_open = False
+            self.app.ui.loop.widget = self.frame
+
+        buttons = urwid.Columns([
+            (urwid.WEIGHT, 0.45, urwid.Button("Yes", on_press=confirm)),
+            (urwid.WEIGHT, 0.1, urwid.Text("")),
+            (urwid.WEIGHT, 0.45, urwid.Button("No", on_press=cancel)),
+        ])
+        try: buttons.focus_position = 2  # default to "No"
+        except Exception: pass
+
+        pile = urwid.Pile([
+            urwid.Text("Are you sure you want to quit?", align="center"),
+            urwid.Divider(),
+            buttons,
+            urwid.Divider(),
+            dont_ask_again,
+        ])
+
+        dialog = QuitDialogLineBox(
+            urwid.Filler(pile, urwid.TOP),
+            title="Quit NomadNet",
+            on_cancel=cancel,
+        )
+
+        self._quit_dialog_open = True
+
+        self.app.ui.loop.widget = urwid.Overlay(
+            dialog, self.frame,
+            align="center", width=44,
+            valign="middle", height=7, min_width=20,
+        )
+
+    def show_notice(self, text):
+        def close(button=None): self.app.ui.loop.widget = self.frame
+        pile = urwid.Pile([urwid.Text(text, align="center"), urwid.Divider(), urwid.Button("OK", on_press=close)])
+        dialog = QuitDialogLineBox(urwid.Filler(pile, urwid.TOP), title="Notice", on_cancel=close)
+        self.app.ui.loop.widget = urwid.Overlay(dialog, self.frame, align="center", width=64, valign="middle", height=9, min_width=20)
+
+    def do_quit(self, sender=None):
         logterm_pid = None
         if True or RNS.vendor.platformutils.is_android():
             if self.sub_displays.log_display != None and self.sub_displays.log_display.log_term != None:
@@ -181,19 +291,33 @@ class MenuDisplay():
         menu_text             = (urwid.PACK, self.menu_indicator)
         button_network        = (11, MenuButton("Network", on_press=handler.show_network))
         button_conversations  = (17, MenuButton("Conversations", on_press=handler.show_conversations))
+        button_channels       = (12, MenuButton("Channels", on_press=handler.show_channels))
         button_directory      = (13, MenuButton("Directory", on_press=handler.show_directory))
         button_map            = (7,  MenuButton("Map", on_press=handler.show_map))
         button_log            = (7,  MenuButton("Log", on_press=handler.show_log))
         button_config         = (10, MenuButton("Config", on_press=handler.show_config))
         button_interfaces     = (14, MenuButton("Interfaces", on_press=handler.show_interfaces))
+        button_utilities      = (13, MenuButton("Utilities", on_press=handler.show_utilities))
         button_guide          = (9,  MenuButton("Guide", on_press=handler.show_guide))
+        button_mynode         = (11, MenuButton("My Node", on_press=handler.show_page_editor))
         button_quit           = (8,  MenuButton("Quit", on_press=handler.quit))
 
         # buttons = [menu_text, button_conversations, button_node, button_directory, button_map]
-        if self.app.config["textui"]["hide_guide"]:
+        buttons = [menu_text, button_conversations, button_network]
-            buttons = [menu_text, button_conversations, button_network, button_log, button_interfaces, button_config, button_quit]
+        if self.app.enable_node:
-        else:
+            buttons.append(button_mynode)
-            buttons = [menu_text, button_conversations, button_network, button_log, button_interfaces, button_config, button_guide, button_quit]
+        buttons += [button_channels, button_log, button_interfaces]
+        enable_utilities = False
+        try:
+            enable_utilities = self.app.config["utilities"].as_bool("enable_utilities")
+        except Exception:
+            enable_utilities = False
+        if enable_utilities:
+            buttons.append(button_utilities)
+        buttons.append(button_config)
+        if not self.app.config["textui"]["hide_guide"]:
+            buttons.append(button_guide)
+        buttons.append(button_quit)
 
         columns = MenuColumns(buttons, dividechars=1)
         columns.handler = handler

nomadnet/ui/textui/MicronParser.py

@@ -1,9 +1,16 @@
 import nomadnet
 import urwid
 import random
+import re
 import time
+import RNS
 from urwid.util import is_mouse_press
 from urwid.text_layout import calc_coords
+from .ReadlineEdit import ReadlineEdit
+from RNS.Utilities.rngit.util import MarkdownToMicron
+from nomadnet.util import STRIP_CONTROL_RE
+try: from .images import ImageWidget
+except Exception: ImageWidget = None
 
 DEFAULT_FG_DARK  = "ddd"
 DEFAULT_FG_LIGHT = "222"
@@ -30,12 +37,28 @@ SYNTH_SPECS  = {}
 
 SECTION_INDENT = 2
 INDENT_RIGHT   = 1
+MAX_TABLE_WIDTH = 100
+
+DEFAULT_FOLD_GLYPHS = ("▾", "▸")
+
+def default_fold_glyphs():
+    try:
+        g = nomadnet.NomadNetworkApp.get_shared_instance().ui.glyphs
+        return (g.get("fold_open", DEFAULT_FOLD_GLYPHS[0]), g.get("fold_closed", DEFAULT_FOLD_GLYPHS[1]))
+    except Exception:
+        return DEFAULT_FOLD_GLYPHS
 
 def default_state(fg=None, bg=None):
+    global SELECTED_STYLES
+    ensure_selected_styles()
     if fg == None: fg = SELECTED_STYLES["plain"]["fg"]
     if bg == None: bg = DEFAULT_BG
     state = {
         "literal": False,
+        "table_mode": False,
+        "table_buffer": [],
+        "table_align": None,
+        "table_maxwidth": MAX_TABLE_WIDTH,
         "depth": 0,
         "fg_color": fg,
         "bg_color": bg,
@@ -50,23 +73,56 @@ def default_state(fg=None, bg=None):
         "align": "left",
         "default_fg": fg,
         "default_bg": bg,
+        "anchors": {},
+        "pending_anchors": [],
+        "header_rows": [],
     }
     return state
 
-def markup_to_attrmaps(markup, url_delegate = None, fg_color=None, bg_color=None):
-    global SELECTED_STYLES
-    if nomadnet.NomadNetworkApp.get_shared_instance().config["textui"]["theme"] == nomadnet.ui.TextUI.THEME_DARK:
-        SELECTED_STYLES = STYLES_DARK
-    else:
-        SELECTED_STYLES = STYLES_LIGHT
 
-    attrmaps = []
+_MICRON_STRIP_RE = re.compile(
+    r"`[FB]T[0-9a-fA-F]{6}"
+    r"|`[FB][0-9a-fA-F]{3}"
+    r"|`:[A-Za-z0-9_\-]*"
+    r"|`[!*_=fbacrl`<>{]"
+)
+
+def slugify_micron(text):
+    if text is None: return ""
+    stripped = _MICRON_STRIP_RE.sub("", text)
+    s = re.sub(r"[^A-Za-z0-9]+", "-", stripped).strip("-").lower()
+    return s
+
+class _AttrMapList(list):
+    """list subclass that allows attaching parser metadata like `anchors`."""
+    pass
+
+def ensure_selected_styles():
+    global SELECTED_STYLES
+    theme = nomadnet.NomadNetworkApp.get_shared_instance().config["textui"]["theme"]
+    if theme == nomadnet.ui.TextUI.THEME_DARK: SELECTED_STYLES = STYLES_DARK
+    else:                                      SELECTED_STYLES = STYLES_LIGHT
+
+def markup_to_attrmaps(markup, url_delegate = None, fg_color=None, bg_color=None, link_class=None, anchors=None):
+    markup = STRIP_CONTROL_RE.sub("", markup)
+    global LINK_CLASS
+    if link_class: LINK_CLASS = link_class
+    else: LINK_CLASS = LinkableText
+
+    global SELECTED_STYLES
+    ensure_selected_styles()
+
+    attrmaps = _AttrMapList()
 
     fgc = None; bgc = DEFAULT_BG
     if bg_color != None: bgc = bg_color
     if fg_color != None: fgc = fg_color
 
     state = default_state(fgc, bgc)
+    if anchors is None:
+        anchors = state["anchors"]
+    else:
+        state["anchors"] = anchors
 
     # Split entire document into lines for
     # processing.
@@ -79,12 +135,344 @@ def markup_to_attrmaps(markup, url_delegate = None, fg_color=None, bg_color=None
             display_widgets = [urwid.Text("")]
 
         if display_widgets != None and len(display_widgets) != 0:
+            row_index = len(attrmaps)
+            pending = state.get("pending_anchors") or []
+            if pending:
+                for name in pending:
+                    if name and name not in anchors:
+                        anchors[name] = row_index
+                state["pending_anchors"] = []
+
+            if state.get("_header_pending"):
+                state["header_rows"].append(row_index)
+                state.setdefault("header_levels", {})[row_index] = state["depth"]
+                cp = state.get("_collapsible_pending")
+                if cp is not None:
+                    state.setdefault("collapsibles", {})[row_index] = bool(cp["collapsed"])
+                    state["_collapsible_pending"] = None
+                state["_header_pending"] = False
+
+            row_levels = state.setdefault("row_levels", {})
             for display_widget in display_widgets:
+                row_levels[len(attrmaps)] = state["depth"]
                 attrmap = urwid.AttrMap(display_widget, make_style(state))
                 attrmaps.append(attrmap)
 
+    try:
+        setattr(attrmaps, "anchors", anchors)
+        setattr(attrmaps, "header_rows", list(state.get("header_rows", [])))
+        setattr(attrmaps, "header_levels", dict(state.get("header_levels", {})))
+        setattr(attrmaps, "row_levels", dict(state.get("row_levels", {})))
+        setattr(attrmaps, "collapsibles", dict(state.get("collapsibles", {})))
+        setattr(attrmaps, "fold_glyphs", state.get("fold_glyphs") or default_fold_glyphs())
+    except Exception:
+        pass
+
     return attrmaps
 
+def parse_partial(line):
+    try:
+        endpos = line.find("}")
+        if endpos == -1: return None
+        else:
+            partial_data = line[0:endpos]
+
+            partial_id = None
+            partial_components = partial_data.split("`")
+            if len(partial_components) == 1:
+                partial_url = partial_components[0]
+                partial_refresh = None
+                partial_fields = ""
+            elif len(partial_components) == 2:
+                partial_url = partial_components[0]
+                partial_refresh = float(partial_components[1])
+                partial_fields = ""
+            elif len(partial_components) == 3:
+                partial_url = partial_components[0]
+                partial_refresh = float(partial_components[1])
+                partial_fields = partial_components[2]
+            else:
+                partial_url = ""
+                partial_fields = ""
+                partial_refresh = None
+
+            if partial_refresh != None and partial_refresh < 1: partial_refresh = None
+
+            pf = partial_fields.split("|")
+            if len(pf) > 0:
+                partial_fields = pf
+                for f in pf:
+                    if f.startswith("pid="):
+                        pcs = f.split("=")
+                        partial_id = pcs[1]
+
+            if len(partial_url):
+                pile = urwid.Pile([urwid.Text(f"⧖")])
+                partial_descriptor = "|".join(partial_components)
+                pile.partial_id = partial_id
+                pile.partial_hash = RNS.hexrep(RNS.Identity.full_hash(partial_descriptor.encode("utf-8")), delimit=False)
+                pile.partial_url = partial_url
+                pile.partial_fields = partial_fields
+                pile.partial_refresh = partial_refresh
+                return [pile]
+
+    except Exception as e: return None
+
+def parse_image(line, state, url_delegate):
+    try:
+        endpos = line.rfind(")")
+        if endpos <= 0: return None
+        image_data = line[:endpos]
+        fields = image_data.split("`")
+        if len(fields) < 2: return None
+
+        alt_text = fields[0].strip()
+        image_url = fields[-1].strip()
+        properties = fields[1:-1]
+
+        width = None
+        height = None
+        align = None
+        for prop in properties:
+            if "=" not in prop: continue
+            key, _, value = prop.partition("=")
+            key = key.strip()
+            value = value.strip()
+            if   key == "w": width = value
+            elif key == "h": height = value
+            elif key == "a": align = value
+
+        if align == "c": align = "center"
+        if align == "l": align = "left"
+        if align == "r": align = "right"
+
+        widget = None
+        if image_url:
+            try: path = url_delegate.resolve_image(image_url)
+            except Exception as e:
+                RNS.trace_exception(e) # TODO: Remove
+                path = None
+            
+            if ImageWidget is not None:
+                try:
+                    if not ImageWidget.placeholder_glyph: ImageWidget.placeholder_glyph = f"{url_delegate.g['image']} "
+
+                    widget = ImageWidget(path, name=alt_text)
+                    for prop_name, prop_value in ( ("width", width), ("height", height), ("align", align) ):
+                        if prop_value is not None:
+                            try:
+                                if isinstance(prop_value, str) and prop_value.isdigit(): prop_value = int(prop_value)
+                                setattr(widget, prop_name, prop_value)
+                            except Exception as e:
+                                RNS.trace_exception(e) # TODO: Remove
+                                pass
+
+                except Exception as e:
+                    RNS.trace_exception(e) # TODO: Remove
+                    widget = None
+
+        if widget is None:
+            placeholder = alt_text if alt_text else (image_url or "image")
+            widget = urwid.Text(f"[{url_delegate.g['warning']} Error loading image: {placeholder}]")
+
+        widget.image_url = image_url
+        widget.image_alt = alt_text
+        if state["depth"] == 0: return [widget]
+        else:
+            padded = urwid.Padding(widget, left=left_indent(state), right=right_indent(state))
+            padded._contained_image = widget
+            return [padded]
+    
+    except Exception as e:
+        # TODO: Remove
+        RNS.trace_exception(e)
+        return None
+
+def render_table(lines, state, url_delegate):
+    if len(lines) < 2: return None
+    if state["table_maxwidth"]: max_width = state["table_maxwidth"]
+    else:                       max_width = MAX_TABLE_WIDTH
+    if state["table_align"]:    align     = state["table_align"]
+    else:                       align     = None
+
+    converter = MarkdownToMicron(max_width=max_width)
+    micron_lines = converter.format_table_raw(lines, align=align)
+
+    widgets = []
+    for line in micron_lines:
+        # Disable table mode to avoid recursion
+        was_table_mode = state["table_mode"]
+        state["table_mode"] = False
+        
+        line_widgets = parse_line(line, state, url_delegate)
+        state["table_mode"] = was_table_mode
+        
+        if line_widgets: widgets.extend(line_widgets)
+    
+    return widgets if widgets else None
+
+class FormColumns(urwid.Columns):
+    MIN_FIELD_WIDTH = 4
+    
+    @staticmethod
+    def _natural_width(widget, maxcol, focused):
+        # mirror the width urwid.Columns itself computes for a PACK column
+        if not isinstance(widget, urwid.Widget):
+            return 0
+        sizing = widget.sizing()
+        if sizing & frozenset((urwid.FIXED, urwid.FLOW)):
+            width = 0
+            if urwid.FIXED in sizing:
+                width = widget.pack((), focused)[0]
+            if urwid.FLOW in sizing and (not width or width > maxcol):
+                width = widget.pack((maxcol,), focused)[0]
+            return width
+        return widget.pack((maxcol,), focused)[0]
+
+    def column_widths(self, size, focus=False):
+        contents = self.contents
+        field_idxs = [i for i, (w, (t, n, b)) in enumerate(contents) if t == urwid.GIVEN]
+
+        if not field_idxs:
+            return super().column_widths(size, focus)
+
+        maxcol = size[0]
+        nat = []
+        for i, (w, (t, width, b)) in enumerate(contents):
+            if t == urwid.GIVEN:
+                nat.append(width)
+            elif t == urwid.PACK:
+                nat.append(self._natural_width(w, maxcol, focus and i == self.focus_position))
+            else:
+                return super().column_widths(size, focus)
+
+        dividers = self.dividechars * max(0, len(nat) - 1)
+        field_set = set(field_idxs)
+        fixed = sum(w for i, w in enumerate(nat) if i not in field_set)
+        requested = sum(nat[i] for i in field_idxs)
+        available = maxcol - dividers - fixed
+
+        if available >= requested:
+            return super().column_widths(size, focus)
+
+        if available < self.MIN_FIELD_WIDTH * len(field_idxs):
+            return super().column_widths(size, focus)
+
+        overflow = requested - available
+        slack = {i: max(0, nat[i] - self.MIN_FIELD_WIDTH) for i in field_idxs}
+        total_slack = sum(slack.values())
+        if total_slack < overflow:
+            return super().column_widths(size, focus)
+
+        widths = list(nat)
+        removed = {i: min(slack[i], overflow * slack[i] // total_slack) for i in field_idxs}
+        shortfall = overflow - sum(removed.values())
+        # Hand out any rounding remainder one column at a time.
+        pending = [i for i in field_idxs if removed[i] < slack[i]]
+        k = 0
+        while shortfall > 0 and pending:
+            i = pending[k % len(pending)]
+            removed[i] += 1
+            shortfall -= 1
+            if removed[i] >= slack[i]:
+                pending = [j for j in pending if removed[j] < slack[j]]
+                k = 0
+            else:
+                k += 1
+        for i in field_idxs:
+            widths[i] = nat[i] - removed[i]
+        return widths
+
+
+class MicronField(ReadlineEdit):
+    def __init__(self, *args, min_rows=1, **kwargs):
+        try:    self._min_rows = max(1, int(min_rows))
+        except (TypeError, ValueError): self._min_rows = 1
+        super().__init__(*args, **kwargs)
+
+    def rows(self, size, focus=False):
+        return max(self._min_rows, super().rows(size, focus))
+
+    def render(self, size, focus=False):
+        canv = super().render(size, focus)
+        if canv.rows() < self._min_rows:
+            canv = urwid.CompositeCanvas(canv)
+            canv.pad_trim_top_bottom(0, self._min_rows - canv.rows())
+        return canv
+
+
+class CollapsibleHeading(urwid.WidgetWrap):
+    _selectable = True
+    ignore_focus = False
+
+    def __init__(self, segments, align, heading_style, depth,
+                 collapsed=False, glyphs=None, delegate=None):
+        self._segments = list(segments)
+        self.align = align
+        self.heading_style = heading_style
+        self.depth = depth
+        self.collapsed = bool(collapsed)
+        self.glyphs = glyphs if glyphs is not None else default_fold_glyphs()
+        self.delegate = delegate
+        self.is_collapsible_heading = True
+        super().__init__(self._build())
+
+    def selectable(self):
+        return True
+
+    def _glyph(self):
+        return self.glyphs[1] if self.collapsed else self.glyphs[0]
+
+    def _glyph_col(self):
+        if self._segments and isinstance(self._segments[0], str):
+            return len(self._segments[0])
+        return 0
+
+    def _build(self):
+        segs = list(self._segments)
+        marker = self._glyph() + " "
+        if segs and isinstance(segs[0], str):
+            segs.insert(1, marker)
+        else:
+            segs.insert(0, marker)
+        return urwid.AttrMap(urwid.Text(segs, align=self.align), self.heading_style)
+
+    def set_collapsed(self, collapsed):
+        collapsed = bool(collapsed)
+        if collapsed != self.collapsed:
+            self.collapsed = collapsed
+            self._w = self._build()
+            self._invalidate()
+
+    def toggle(self):
+        self.collapsed = not self.collapsed
+        self._w = self._build()
+        self._invalidate()
+        if self.delegate is not None and hasattr(self.delegate, "fold_changed"):
+            self.delegate.fold_changed(self)
+
+    def keypress(self, size, key):
+        if self._command_map[key] == urwid.ACTIVATE or key == " ":
+            self.toggle()
+            return None
+        return key
+
+    def mouse_event(self, size, event, button, x, y, focus):
+        if button == 1 and is_mouse_press(event):
+            self.toggle()
+            return True
+        return False
+
+    def render(self, size, focus=False):
+        canv = super().render(size, focus)
+        if focus:
+            canv = urwid.CompositeCanvas(canv)
+            gx = self._glyph_col()
+            if self.align == "left" and gx < size[0]:
+                canv.cursor = (gx, 0)
+        return canv
+
+
 def parse_line(line, state, url_delegate):
     pre_escape = False
     if len(line) > 0:
@@ -97,6 +485,20 @@ def parse_line(line, state, url_delegate):
 
         # Only parse content if not in literal state
         if not state["literal"]:
+            collapsible_heading = False
+            collapsed_initial = False
+            if (line.startswith("`+") or line.startswith("`-")) and line[2:3] == ">":
+                collapsible_heading = True
+                collapsed_initial = line[1] == "-"
+                line = line[2:]
+                first_char = line[0]
+
+            # Apply markup sanitization
+            if first_char == ">" and "`<" in line:
+                # Remove heading status from lines containing fields
+                line = line.lstrip(">")
+                first_char = line[0]
+
             # Check if the command is an escape
             if first_char == "\\":
                 line = line[1:]
@@ -104,8 +506,50 @@ def parse_line(line, state, url_delegate):
 
             # Check for comments
             elif first_char == "#":
+                parts = line.split()
+                if parts and parts[0] == "#!fold":
+                    if len(parts) >= 3:   state["fold_glyphs"] = (parts[1], parts[2])
+                    elif len(parts) == 2: state["fold_glyphs"] = (parts[1], parts[1])
                 return None
 
+            # Check for tables
+            if line.startswith("`t"):
+                line = line[2:]
+                align = line[0] if len(line) and line[0] in ["l", "c", "r"] else None
+                max_width = None
+                if align: line = line[1:]
+                if len(line):
+                    try: max_width = int(line)
+                    except: pass
+
+                if state["table_mode"]:
+                    widgets = render_table(state["table_buffer"], state, url_delegate)
+                    state["table_mode"] = False
+                    state["table_buffer"] = []
+                    state["table_align"] = None
+                    state["table_maxwidth"] = MAX_TABLE_WIDTH
+                    return widgets
+
+                else:
+                    state["table_mode"] = True
+                    state["table_buffer"] = []
+                    state["table_align"] = align
+                    state["table_maxwidth"] = max_width
+                    return None
+
+            # Buffer the line if in table mode
+            if state["table_mode"]:
+                state["table_buffer"].append(line)
+                return None
+
+            # Check for partials
+            elif line.startswith("`{"):
+                return parse_partial(line[2:])
+
+            # Check for images
+            elif line.startswith("`("):
+                return parse_image(line[2:], state, url_delegate)
+
             # Check for section heading reset
             elif first_char == "<":
                 state["depth"] = 0
@@ -133,11 +577,21 @@ def parse_line(line, state, url_delegate):
 
                     style_to_state(latched_style, state)
 
+                    slug = slugify_micron(line)
+                    if slug:
+                        state.setdefault("pending_anchors", []).append(slug)
+                    state["_header_pending"] = True
+
                     if len(output) > 0:
                         first_style = output[0][0]
 
                         heading_style = first_style
                         output.insert(0, " "*left_indent(state))
+                        if collapsible_heading:
+                            state["_collapsible_pending"] = {"collapsed": collapsed_initial}
+                            glyphs = state.get("fold_glyphs") or default_fold_glyphs()
+                            return [CollapsibleHeading(output, state["align"], heading_style,
+                                                       state["depth"], collapsed_initial, glyphs, url_delegate)]
                         return [urwid.AttrMap(urwid.Text(output, align=state["align"]), heading_style)]
                     else:
                         return None
@@ -149,14 +603,14 @@ def parse_line(line, state, url_delegate):
                 if len(line) == 2:
                     divider_char = line[1]
                     # Control characters don't make sense here and otherwise crash nomadnet
-                    if ord(divider_char) < 32:
+                    if ord(divider_char) < 32: divider_char = "\u2500"
-                        divider_char = "\u2500"
+                    # Check if the divider char is renderable
-                else:
+                    try: urwid.Divider(divider_char).render((30,))
-                    divider_char = "\u2500"
+                    except Exception as e: divider_char = "\u2500"
-                if state["depth"] == 0:
+                else: divider_char = "\u2500"
-                    return [urwid.Divider(divider_char)]
+
-                else:
+                if state["depth"] == 0: return [urwid.Divider(divider_char)]
-                    return [urwid.Padding(urwid.Divider(divider_char), left=left_indent(state), right=right_indent(state))]
+                else:                   return [urwid.Padding(urwid.Divider(divider_char), left=left_indent(state), right=right_indent(state))]
 
         output = make_output(state, line, url_delegate, pre_escape)
 
@@ -172,7 +626,7 @@ def parse_line(line, state, url_delegate):
                 for o in output:
                     if isinstance(o, tuple):
                         if url_delegate != None:
-                            tw = LinkableText(o, align=state["align"], delegate=url_delegate)
+                            tw = LINK_CLASS(o, align=state["align"], delegate=url_delegate)
                             tw.in_columns = True
                         else:
                             tw = urwid.Text(o, align=state["align"])
@@ -183,8 +637,9 @@ def parse_line(line, state, url_delegate):
                             fd = o["data"]
                             fn = o["name"]
                             fs = o["style"]
+                            frows = o.get("rows", 1)
                             fmask = "*" if o["masked"] else None
-                            f = urwid.Edit(caption="", edit_text=fd, align=state["align"], multiline=True, mask=fmask)
+                            f = MicronField(caption="", edit_text=fd, align=state["align"], multiline=True, mask=fmask, min_rows=frows)
                             f.field_name = fn
                             fa = urwid.AttrMap(f, fs)
                             widgets.append((fw, fa))
@@ -219,13 +674,13 @@ def parse_line(line, state, url_delegate):
 
 
 
-                columns_widget = urwid.Columns(widgets, dividechars=0)
+                columns_widget = FormColumns(widgets, dividechars=0)
                 text_widget = columns_widget
                 # text_widget = urwid.Text("<"+output+">", align=state["align"])
 
             else:
                 if url_delegate != None:
-                    text_widget = LinkableText(output, align=state["align"], delegate=url_delegate)
+                    text_widget = LINK_CLASS(output, align=state["align"], delegate=url_delegate)
                 else:
                     text_widget = urwid.Text(output, align=state["align"])
 
@@ -283,14 +738,11 @@ def make_style(state):
 
                 if color[0] == "g":
                     val = int(color[1:2])
-                    if val < 25:
+                    if val < 25:   result = "black"
-                        result = "black"
+                    elif val < 50: result = "dark gray"
-                    elif val < 50:
+                    elif val < 75: result = "light gray"
-                        result = "dark gray"
+                    else:          result = "white"
-                    elif val < 75:
+                
-                        result = "light gray"
-                    else:
-                        result = "white"
                 else:
                     r = int(color[0], 16)
                     g = int(color[1], 16)
@@ -298,65 +750,43 @@ def make_style(state):
 
                     if r == g == b:
                         val = int(color[0], 16)*6
-                        if val < 12:
+                        if val < 12:   result = "black"
-                            result = "black"
+                        elif val < 50: result = "dark gray"
-                        elif val < 50:
+                        elif val < 80: result = "light gray"
-                            result = "dark gray"
+                        else:          result = "white"
-                        elif val < 80:
-                            result = "light gray"
-                        else:
-                            result = "white"
 
                     else:
                         if r == b:
                             if r > g:
-                                if r > t:
+                                if r > t: result = "light magenta"
-                                    result = "light magenta"
+                                else:     result = "dark magenta"
-                                else:
-                                    result = "dark magenta"
                             else:
-                                if g > t:
+                                if g > t: result = "light green"
-                                    result = "light green"
+                                else:     result = "dark green"
-                                else:
-                                    result = "dark green"
                         if b == g:
                             if b > r:
-                                if b > t:
+                                if b > t: result = "light cyan"
-                                    result = "light cyan"
+                                else:     result = "dark cyan"
-                                else:
-                                    result = "dark cyan"
                             else:
-                                if r > t:
+                                if r > t: result = "light red"
-                                    result = "light red"
+                                else:     result = "dark red"
-                                else:
-                                    result = "dark red"
                         if g == r:
                             if g > b:
-                                if g > t:
+                                if g > t: result = "yellow"
-                                    result = "yellow"
+                                else:     result = "brown"
-                                else:
-                                    result = "brown"
                             else:
-                                if b > t:
+                                if b > t: result = "light blue"
-                                    result = "light blue"
+                                else:     result = "dark blue"
-                                else:
-                                    result = "dark blue"
 
                         if r > g and r > b:
-                            if r > t:
+                            if r > t: result = "light red"
-                                result = "light red"
+                            else:     result = "dark red"
-                            else:
-                                result = "dark red"
                         if g > r and g > b:
-                            if g > t:
+                            if g > t: result = "light green"
-                                result = "light green"
+                            else:     result = "dark green"
-                            else:
-                                result = "dark green"
                         if b > g and b > r:
-                            if b > t:
+                            if b > t: result = "light blue"
-                                result = "light blue"
+                            else:     result = "dark blue"
-                            else:
-                                result = "dark blue"
 
         except Exception as e:
             result = "default"
@@ -422,12 +852,9 @@ def make_style(state):
     bg        = state["bg_color"]
 
     format_string = ""
-    if bold:
+    if bold:      format_string += ",bold"
-        format_string += ",bold"
+    if underline: format_string += ",underline"
-    if underline:
+    if italic:    format_string += ",italics"
-        format_string += ",underline"
-    if italic:
-        format_string += ",italics"
 
     name = "micron_"+fg+"_"+bg+"_"+format_string
     if not name in SYNTH_STYLES:
@@ -467,16 +894,26 @@ def make_output(state, line, url_delegate, pre_escape=False):
                         state["formatting"]["italic"] ^= True
                     elif c == "F":
                         if len(line) >= i+4:
-                            color = line[i+1:i+4]
+                            if line[i+1] == "T" and len(line) >= i+8:
-                            state["fg_color"] = color
+                                color = line[i+2:i+8]
-                            skip = 3
+                                state["fg_color"] = color
+                                skip = 7
+                            else:
+                                color = line[i+1:i+4]
+                                state["fg_color"] = color
+                                skip = 3
                     elif c == "f":
                         state["fg_color"] = state["default_fg"]
                     elif c == "B":
                         if len(line) >= i+4:
-                            color = line[i+1:i+4]
+                            if line[i+1] == "T" and len(line) >= i+8:
-                            state["bg_color"] = color
+                                color = line[i+2:i+8]
-                            skip = 3
+                                state["bg_color"] = color
+                                skip = 7
+                            else:
+                                color = line[i+1:i+4]
+                                state["bg_color"] = color
+                                skip = 3
                     elif c == "b":
                         state["bg_color"] = state["default_bg"]
                     elif c == "`":
@@ -488,16 +925,25 @@ def make_output(state, line, url_delegate, pre_escape=False):
                         state["align"] = state["default_align"]
                     elif c == "c":
                         if state["align"] != "center": state["align"] = "center"
-                        # else: state["align"] = state["default_align"]
                     elif c == "l":
                         if state["align"] != "left": state["align"] = "left"
-                        # else: state["align"] = state["default_align"]
                     elif c == "r":
                         if state["align"] != "right": state["align"] = "right"
-                        # else: state["align"] = state["default_align"]
                     elif c == "a":
                         state["align"] = state["default_align"]
 
+                    elif c == ":":
+                        # Anchor declaration: `:anchor-name  (terminated by any non-name char). The anchor is a zero width position narker bound to the current line by markup_to_attrmaps
+                        name_start = i + 1
+                        name_end = name_start
+                        while name_end < len(line) and (line[name_end].isalnum() or line[name_end] in "_-"):
+                            name_end += 1
+                        anchor_name = line[name_start:name_end]
+                        if anchor_name:
+                            state.setdefault("pending_anchors", []).append(anchor_name)
+                        skip = (name_end - i) - 1
+                        if skip < 0: skip = 0
+
                     elif c == '<':
                         if len(part) > 0:
                             output.append(make_part(state, part))
@@ -511,6 +957,7 @@ def make_output(state, line, url_delegate, pre_escape=False):
                                 field_content = line[field_start:backtick_pos]
                                 field_masked = False
                                 field_width = 24
+                                field_rows = 1
                                 field_type = "field"
                                 field_name = field_content
                                 field_value = ""
@@ -536,10 +983,16 @@ def make_output(state, line, url_delegate, pre_escape=False):
 
                                     # Handle field width
                                     if len(field_flags) > 0:
+                                        width_flag, _, rows_flag = field_flags.partition("x")
                                         try:
-                                            field_width = min(int(field_flags), 256)
+                                            field_width = min(int(width_flag), 256)
                                         except ValueError:
                                             pass  # Ignore invalid width
+                                        if rows_flag:
+                                            try:
+                                                field_rows = max(1, min(int(rows_flag), 256))
+                                            except ValueError:
+                                                pass
 
                                     # Check for value and pre-checked flag
                                     if len(f_components) > 2:
@@ -583,6 +1036,7 @@ def make_output(state, line, url_delegate, pre_escape=False):
                                             "type": "field",
                                             "name": field_name,
                                             "width": field_width,
+                                            "rows": field_rows,
                                             "masked": field_masked,
                                             "data": field_data,
                                             "style": make_style(state)
@@ -653,8 +1107,6 @@ def make_output(state, line, url_delegate, pre_escape=False):
                                 else:
                                     output.append(make_part(state, link_label))
 
-                                
-
                     mode = "text"
                     if len(part) > 0:
                         output.append(make_part(state, part))
@@ -707,7 +1159,9 @@ class LinkableText(urwid.Text):
         super().__init__(text, align=align)
         self.delegate = delegate
         self._cursor_position = 0
-        self.key_timeout = 3
+        self._cursor_promised = None
+        self._cursor_promised_at = 0
+        self.key_timeout = 2
         self.in_columns = False
         if self.delegate != None:
             self.delegate.last_keypress = 0
@@ -746,7 +1200,7 @@ class LinkableText(urwid.Text):
         if item != None:
             if isinstance(item, LinkSpec):
                 if self.delegate != None:
-                    self.delegate.marked_link(item.link_target)
+                    self.delegate.marked_link(item.link_target, item.link_fields)
             else:
                 if self.delegate != None:
                     self.delegate.marked_link(None)
@@ -813,28 +1267,35 @@ class LinkableText(urwid.Text):
     def kt_event(self, loop, user_data):
         self._invalidate()
 
+    def cursor_visible(self):
+        return self.delegate == None or time.time() < self.delegate.last_keypress+self.key_timeout
+
     def render(self, size, focus=False):
-        now = time.time()
         c = super().render(size, focus)
-
+        if focus:
-        if focus and (self.delegate == None or now < self.delegate.last_keypress+self.key_timeout):
+            if time.time() < self._cursor_promised_at+0.5:
-            c = urwid.CompositeCanvas(c)
+                coords = self._cursor_promised
-            c.cursor = self.get_cursor_coords(size)
+            else:
-            if self.delegate != None:
+                coords = self.get_cursor_coords(size)
-                self.peek_link()
+            if coords != None:
-
+                c = urwid.CompositeCanvas(c)
+                c.cursor = coords
+                if self.delegate != None:
+                    self.peek_link()
         return c
 
     def get_cursor_coords(self, size):
-            if self._cursor_position > len(self.text):
+        self._cursor_promised_at = time.time()
-                return None
+        self._cursor_promised = None
-
+        if not self.cursor_visible() or self._cursor_position > len(self.text):
-            (maxcol,) = size
+            return None
-            trans = self.get_line_translation(maxcol)
+        (maxcol,) = size
-            x, y = calc_coords(self.text, trans, self._cursor_position)
+        trans = self.get_line_translation(maxcol)
-            if maxcol <= x:
+        x, y = calc_coords(self.text, trans, self._cursor_position)
-                return None
+        if maxcol <= x:
-            return x, y
+            return None
+        self._cursor_promised = (x, y)
+        return x, y
 
     def mouse_event(self, size, event, button, x, y, focus):
         try:
@@ -879,3 +1340,5 @@ class LinkableText(urwid.Text):
             
         except Exception as e:
             return False
+
+LINK_CLASS = LinkableText

nomadnet/ui/textui/Network.py

@@ -6,20 +6,47 @@ import threading
 from datetime import datetime
 from nomadnet.Directory import DirectoryEntry
 from nomadnet.vendor.additional_urwid_widgets import IndicativeListBox, MODIFIER_KEY
+from nomadnet.util import strip_modifiers
+from nomadnet.util import sanitize_name
 
 from .Browser import Browser
+from .ReadlineEdit import ReadlineEdit
+from .Helpers import qr_ascii
 
 class NetworkDisplayShortcuts():
     def __init__(self, app):
         self.app = app
         g = app.ui.glyphs
 
-        self.widget = urwid.AttrMap(urwid.Text("[C-l] Nodes/Announces  [C-x] Remove  [C-w] Disconnect  [C-d] Back  [C-f] Forward  [C-r] Reload  [C-u] URL  [C-g] Fullscreen  [C-s / C-b] Save Node"), "shortcutbar")
+        shortcut_text = "[C-l] Nodes/Announces  [C-x] Remove  [C-g] Fullscreen"
+        self.widget = urwid.AttrMap(urwid.Text(shortcut_text), "shortcutbar")
+
+class BrowserShortcuts():
+    def __init__(self, app, browser):
+        self.app = app
+        self.browser = browser
+        g = app.ui.glyphs
+
+        shortcut_text = "[C-w] Disconnect  [C-d] Back  [C-f] Forward  [C-r] Reload  [C-u] URL  "
+        if app.config["textui"]["clipboard_copy"]: shortcut_text += "[C-y] Copy  "
+        shortcut_text += "[C-g] Fullscreen  [C-s/b] Save Node"
+        if browser.image_rendering_supported: shortcut_text += "  [C-l] Load Images  [C-x] Reload Images"
+        self.widget = urwid.AttrMap(urwid.Text(shortcut_text), "shortcutbar")
 
 class DialogLineBox(urwid.LineBox):
     def keypress(self, size, key):
         if key == "esc":
-            self.delegate.update_conversation_list()
+            delegate = getattr(self, "delegate", None)
+            if delegate is not None and hasattr(delegate, "update_conversation_list"):
+                delegate.update_conversation_list()
+            elif delegate is not None and getattr(delegate, "parent", None) is not None:
+                try:
+                    delegate.dialog_open = False
+                    options = delegate.parent.left_pile.options(height_type=urwid.PACK, height_amount=None)
+                    delegate.parent.left_pile.contents[1] = (type(delegate)(delegate.app, delegate.parent), options)
+                except Exception:
+                    pass
+            return None
         else:
             return super(DialogLineBox, self).keypress(size, key)
 
@@ -75,7 +102,7 @@ class AnnounceInfo(urwid.WidgetWrap):
         is_node = False
         is_pn   = False
         if info_type == "node" or info_type == True:
-            type_string = "Nomad Network Node " + g["node"]
+            type_string = "NomadNet Node " + g["node"]
             is_node = True
         elif info_type == "pn":
             type_string = "LXMF Propagation Node " + g["sent"]
@@ -84,7 +111,7 @@ class AnnounceInfo(urwid.WidgetWrap):
             type_string = "Peer " + g["peer"]
 
         try:
-            data_str = announce[2].decode("utf-8")
+            data_str = strip_modifiers(announce[2].decode("utf-8"))
             data_style = ""
             if trust_level != DirectoryEntry.TRUSTED and len(data_str) > 32:
                 data_str = data_str[:32]+" [...]"
@@ -146,7 +173,7 @@ class AnnounceInfo(urwid.WidgetWrap):
                     display_name = op_str
 
                     if not source_hash_text in [c[0] for c in existing_conversations]:
-                        entry = DirectoryEntry(source_hash, display_name, trust_level)
+                        entry = DirectoryEntry(op_hash, display_name, trust_level)
                         self.app.directory.remember(entry)
 
                         new_conversation = nomadnet.Conversation(source_hash_text, nomadnet.NomadNetworkApp.get_shared_instance(), initiator=True)
@@ -250,7 +277,7 @@ class AnnounceInfo(urwid.WidgetWrap):
 
 
 class AnnounceStreamEntry(urwid.WidgetWrap):
-    def __init__(self, app, announce, delegate):
+    def __init__(self, app, announce, delegate, show_destination=False):
         full_time_format = "%Y-%m-%d %H:%M:%S"
         date_time_format = "%Y-%m-%d"
         time_time_format = "%H:%M:%S"
@@ -274,7 +301,19 @@ class AnnounceStreamEntry(urwid.WidgetWrap):
             ts_string = dt.strftime(date_only_format)
 
         trust_level  = self.app.directory.trust_level(source_hash)
-        display_str = self.app.directory.simplest_display_str(source_hash)
+
+        def san(name):
+            if self.app.config["textui"]["sanitize_names"]: return sanitize_name(name)
+            else:                                           return strip_modifiers(name)
+        
+        if show_destination:
+            display_str = RNS.hexrep(source_hash, delimit=False)
+        else:
+            try:    display_str = san(announce[2].decode("utf-8"))
+            except: display_str = self.app.directory.simplest_display_str(source_hash)
+
+        if not display_str: display_str = RNS.prettyhexrep(source_hash)
+        if len(display_str) > 34: display_str = display_str[:34] + "…"
 
         if trust_level == DirectoryEntry.UNTRUSTED:
             symbol        = g["cross"]
@@ -381,22 +420,33 @@ class AnnounceStream(urwid.WidgetWrap):
         self.ilb = None
         self.no_content = True
         self.current_tab = "nodes"
+        self.show_destination = False
+        self.search_text = ""
 
         self.added_entries = []
         self.widget_list = []
-        self.update_widget_list()
 
-        # Create tab buttons
+        self.tab_nodes = TabButton("Nodes (0)", on_press=self.show_nodes_tab)
-        self.tab_nodes = TabButton("Nodes", on_press=self.show_nodes_tab)
+        self.tab_peers = TabButton("Peers (0)", on_press=self.show_peers_tab)
-        self.tab_peers = TabButton("Peers", on_press=self.show_peers_tab)
+        self.tab_pn = TabButton("Propagation Nodes (0)", on_press=self.show_pn_tab)
-        self.tab_pn = TabButton("Propagation Nodes", on_press=self.show_pn_tab)
 
-        # Create tab bar with proportional widths
         self.tab_bar = urwid.Columns([
             ('weight', 1, self.tab_nodes),
             ('weight', 1, self.tab_peers),
             ('weight', 3, self.tab_pn),
-        ], dividechars=1)  # Add 1 character spacing between tabs
+        ], dividechars=1)
+
+        self.search_edit = ReadlineEdit(caption="Search: ")
+        urwid.connect_signal(self.search_edit, 'change', self.on_search_change)
+
+        self.display_toggle = TabButton("Show: Name", on_press=self.toggle_display_mode)
+
+        self.filter_bar = urwid.Columns([
+            ('weight', 2, self.search_edit),
+            ('weight', 1, self.display_toggle),
+        ], dividechars=1)
+
+        self.update_widget_list()
 
         self.ilb = ExceptionHandlingListBox(
             self.widget_list,
@@ -406,9 +456,9 @@ class AnnounceStream(urwid.WidgetWrap):
             #highlight_offFocus="list_off_focus"
         )
 
-        # Combine tab bar and list box
         self.pile = urwid.Pile([
             ('pack', self.tab_bar),
+            ('pack', self.filter_bar),
             ('weight', 1, self.ilb),
         ])
 
@@ -416,17 +466,31 @@ class AnnounceStream(urwid.WidgetWrap):
         super().__init__(urwid.LineBox(self.display_widget, title="Announce Stream"))
 
     def keypress(self, size, key):
-        if key == "up" and (self.no_content or self.ilb.first_item_is_selected()):
+        if key == "up" and self.pile.focus == self.tab_bar:
             nomadnet.NomadNetworkApp.get_shared_instance().ui.main_display.frame.focus_position = "header"
         elif key == "ctrl x":
             self.delete_selected_entry()
 
         return super(AnnounceStream, self).keypress(size, key)
 
+    def on_search_change(self, widget, text):
+        self.search_text = text.lower()
+        self.update_widget_list()
+
+    def toggle_display_mode(self, button):
+        self.show_destination = not self.show_destination
+        if self.show_destination:
+            self.display_toggle.set_label("Show: Dest")
+        else:
+            self.display_toggle.set_label("Show: Name")
+        self.update_widget_list()
+
     def delete_selected_entry(self):
-        if self.ilb.get_selected_item() != None:
+        sel = self.ilb.get_selected_item()
-            self.app.directory.remove_announce_with_timestamp(self.ilb.get_selected_item().original_widget.timestamp)
+        if sel != None and hasattr(sel, "original_widget") and sel.original_widget:
-            self.rebuild_widget_list()
+            if hasattr(sel.original_widget, "timestamp"):
+                self.app.directory.remove_announce_with_timestamp(sel.original_widget.timestamp)
+                self.rebuild_widget_list()
 
     def rebuild_widget_list(self):
         self.no_content = True
@@ -438,19 +502,35 @@ class AnnounceStream(urwid.WidgetWrap):
         self.widget_list = []
         new_entries = []
 
+        node_count = 0
+        peer_count = 0
+        pn_count = 0
+
         for e in self.app.directory.announce_stream:
             announce_type = e[3]
 
-            # Filter based on current tab
+            if self.search_text:
-            if self.current_tab == "nodes" and (announce_type == "node" or announce_type == True):
+                try:
-                new_entries.append(e)
+                    announce_data = e[2].decode("utf-8").lower()
-            elif self.current_tab == "peers" and (announce_type == "peer" or announce_type == False):
+                except:
-                new_entries.append(e)
+                    announce_data = ""
-            elif self.current_tab == "pn" and announce_type == "pn":
+                if self.search_text not in announce_data:
-                new_entries.append(e)
+                    continue
+
+            if announce_type == "node" or announce_type == True:
+                node_count += 1
+                if self.current_tab == "nodes": new_entries.append(e)
+            
+            elif announce_type == "peer" or announce_type == False:
+                peer_count += 1
+                if self.current_tab == "peers": new_entries.append(e)
+
+            elif announce_type == "pn":
+                pn_count += 1
+                if self.current_tab == "pn": new_entries.append(e)
 
         for e in new_entries:
-            nw = AnnounceStreamEntry(self.app, e, self)
+            nw = AnnounceStreamEntry(self.app, e, self, show_destination=self.show_destination)
             nw.timestamp = e[0]
             self.widget_list.append(nw)
 
@@ -460,6 +540,10 @@ class AnnounceStream(urwid.WidgetWrap):
             self.no_content = True
             self.widget_list = [urwid.Text(f"No {self.current_tab} announces", align='center')]
 
+        self.tab_nodes.set_label(f"Nodes ({node_count})")
+        self.tab_peers.set_label(f"Peers ({peer_count})")
+        self.tab_pn.set_label(f"Propagation Nodes ({pn_count})")
+
         if self.ilb:
             self.ilb.set_body(self.widget_list)
 
@@ -555,7 +639,7 @@ class KnownNodeInfo(urwid.WidgetWrap):
         if node_entry == None:
             display_str = self.app.directory.simplest_display_str(source_hash)
         else:
-            display_str = node_entry.display_name
+            display_str = strip_modifiers(node_entry.display_name)
 
         addr_str     = "<"+RNS.hexrep(source_hash, delimit=False)+">"
 
@@ -570,7 +654,7 @@ class KnownNodeInfo(urwid.WidgetWrap):
             lxmf_addr_str = "No associated Propagation Node known"
 
 
-        type_string = "Nomad Network Node " + g["node"]
+        type_string = "NomadNet Node " + g["node"]
 
         if trust_level == DirectoryEntry.UNTRUSTED:
             trust_str     = "Untrusted"
@@ -611,8 +695,8 @@ class KnownNodeInfo(urwid.WidgetWrap):
         r_unknown   = urwid.RadioButton(trust_button_group, "Unknown", state=unknown_selected)
         r_trusted   = urwid.RadioButton(trust_button_group, "Trusted", state=trusted_selected)
 
-        e_name = urwid.Edit(caption="Name      : ",edit_text=display_str)
+        e_name = ReadlineEdit(caption="Name      : ",edit_text=display_str)
-        e_sort = urwid.Edit(caption="Sort Rank : ",edit_text=sort_str)
+        e_sort = ReadlineEdit(caption="Sort Rank : ",edit_text=sort_str)
 
         node_ident = RNS.Identity.recall(source_hash)
         op_hash = None
@@ -641,7 +725,7 @@ class KnownNodeInfo(urwid.WidgetWrap):
                     display_name = op_str
 
                     if not source_hash_text in [c[0] for c in existing_conversations]:
-                        entry = DirectoryEntry(source_hash, display_name, trust_level)
+                        entry = DirectoryEntry(op_hash, display_name, trust_level)
                         self.app.directory.remember(entry)
 
                         new_conversation = nomadnet.Conversation(source_hash_text, nomadnet.NomadNetworkApp.get_shared_instance(), initiator=True)
@@ -926,7 +1010,7 @@ class NodeEntry(urwid.WidgetWrap):
         g = self.app.ui.glyphs
 
         trust_level  = self.app.directory.trust_level(source_hash)
-        display_str = self.app.directory.simplest_display_str(source_hash)
+        display_str =  self.app.directory.simplest_display_str(source_hash, san=False)
 
         if trust_level == DirectoryEntry.UNTRUSTED:
             symbol        = g["cross"]
@@ -1206,7 +1290,7 @@ class LocalPeer(urwid.WidgetWrap):
 
         t_id =           urwid.Text("LXMF Addr : "+RNS.prettyhexrep(self.app.lxmf_destination.hash))
         i_id =           urwid.Text("Identity  : "+RNS.prettyhexrep(self.app.identity.hash))
-        e_name = urwid.Edit(caption="Name      : ", edit_text=display_name)
+        e_name = ReadlineEdit(caption="Name      : ", edit_text=display_name)
 
         def save_query(sender):
             def dismiss_dialog(sender):
@@ -1267,22 +1351,27 @@ class LocalPeer(urwid.WidgetWrap):
 
         announce_button = urwid.Button("Announce Now", on_press=announce_query)
 
-        self.display_widget = urwid.Pile(
+        peer_widgets = [
-            [
+            t_id,
-                t_id,
+            i_id,
-                i_id,
+            e_name,
-                e_name,
+            urwid.Divider(g["divider1"]),
-                urwid.Divider(g["divider1"]),
+            self.t_last_announce,
-                self.t_last_announce,
+            announce_button,
-                announce_button,
+            urwid.Divider(g["divider1"]),
-                urwid.Divider(g["divider1"]),
+            urwid.Columns([
-                urwid.Columns([
+                (urwid.WEIGHT, 0.45, urwid.Button("Save", on_press=save_query)),
-                    (urwid.WEIGHT, 0.45, urwid.Button("Save", on_press=save_query)),
+                (urwid.WEIGHT, 0.1, urwid.Text("")),
-                    (urwid.WEIGHT, 0.1, urwid.Text("")),
+                (urwid.WEIGHT, 0.45, urwid.Button("Node Info", on_press=node_info_query)),
-                    (urwid.WEIGHT, 0.45, urwid.Button("Node Info", on_press=node_info_query)),
+            ]),
-                ])
+        ]
-            ]
+
-        )
+        if self.app.enable_node:
+            peer_widgets.append(
+                urwid.AttrMap(urwid.Text(" "+g["node"]+" Node Online", align=urwid.LEFT), "shortcutbar")
+            )
+
+        self.display_widget = urwid.Pile(peer_widgets)
 
         super().__init__(urwid.LineBox(self.display_widget, title="Local Peer Info"))
 
@@ -1356,6 +1445,37 @@ class NodeInfo(urwid.WidgetWrap):
             def connect_query(sender):
                 self.parent.browser.retrieve_url(RNS.hexrep(self.app.node.destination.hash, delimit=False))
 
+            def edit_query(sender):
+                self.app.ui.main_display.show_page_editor(None)
+
+            def show_qr_query(sender):
+                def dismiss_dialog(sender):
+                    self.dialog_open = False
+                    options = self.parent.left_pile.options(height_type=urwid.PACK, height_amount=None)
+                    self.parent.left_pile.contents[1] = (NodeInfo(self.app, self.parent), options)
+
+                dest_addr = RNS.hexrep(self.app.node.destination.hash, delimit=False)
+                qr_text = qr_ascii(dest_addr)
+
+                rows = [urwid.Text("")]
+                if qr_text is not None:
+                    rows.append(urwid.Text(qr_text, align=urwid.CENTER))
+                else:
+                    rows.append(urwid.Text("Node destination address:", align=urwid.CENTER))
+                rows += [
+                    urwid.Text(""),
+                    urwid.Text("< "+dest_addr+" >", align=urwid.CENTER),
+                    urwid.Text(""),
+                    urwid.Button("Close", on_press=dismiss_dialog),
+                ]
+
+                dialog = DialogLineBox(urwid.Pile(rows), title="Node Destination")
+                dialog.delegate = self
+
+                self.dialog_open = True
+                options = self.parent.left_pile.options(height_type=urwid.PACK, height_amount=None)
+                self.parent.left_pile.contents[1] = (dialog, options)
+
             if NodeInfo.announce_timer == None:
                 self.t_last_announce = NodeAnnounceTime(self.app)
                 NodeInfo.announce_timer = self.t_last_announce
@@ -1403,7 +1523,9 @@ class NodeInfo(urwid.WidgetWrap):
 
             announce_button = urwid.Button("Announce", on_press=announce_query)
             connect_button = urwid.Button("Browse", on_press=connect_query)
+            edit_button = urwid.Button("Edit", on_press=edit_query)
             reset_button = urwid.Button("Rst Stats", on_press=stats_query)
+            qr_button = urwid.Button("Show Destination", on_press=show_qr_query)
 
             if not self.app.disable_propagation:
                 pile = urwid.Pile([
@@ -1427,6 +1549,11 @@ class NodeInfo(urwid.WidgetWrap):
                         (urwid.WEIGHT, 8, reset_button),
                         (urwid.WEIGHT, 0.5, urwid.Text("")),
                         (urwid.WEIGHT, 7, announce_button),
+                    ]),
+                    urwid.Columns([
+                        (urwid.WEIGHT, 4, edit_button),
+                        (urwid.WEIGHT, 0.5, urwid.Text("")),
+                        (urwid.WEIGHT, 9, qr_button),
                     ])
                 ])
             else:
@@ -1449,6 +1576,11 @@ class NodeInfo(urwid.WidgetWrap):
                     (urwid.WEIGHT, 8, reset_button),
                     (urwid.WEIGHT, 0.5, urwid.Text("")),
                     (urwid.WEIGHT, 7, announce_button),
+                ]),
+                urwid.Columns([
+                    (urwid.WEIGHT, 4, edit_button),
+                    (urwid.WEIGHT, 0.5, urwid.Text("")),
+                    (urwid.WEIGHT, 9, qr_button),
                 ])
             ])
         else:
@@ -1576,10 +1708,11 @@ class NetworkDisplay():
         self.known_nodes_display.delegate = self
 
         self.list_display = 1
+        default_info_display = self.node_info_display if self.app.enable_node else self.local_peer_display
         self.left_pile = NetworkLeftPile([
             (urwid.WEIGHT, 1, self.known_nodes_display),
             # (urwid.PACK, self.network_stats_display),
-            (urwid.PACK, self.local_peer_display),
+            (urwid.PACK, default_info_display),
         ])
 
         self.left_pile.parent = self
@@ -1598,7 +1731,9 @@ class NetworkDisplay():
             dividechars=0, focus_column=0
         )
 
-        self.shortcuts_display = NetworkDisplayShortcuts(self.app)
+        self.list_shortcuts = NetworkDisplayShortcuts(self.app)
+        self.browser_shortcuts = BrowserShortcuts(self.app, self.browser)
+        self.shortcuts_display = self.list_shortcuts
         self.widget = self.columns
 
     def toggle_list(self):
@@ -1648,7 +1783,6 @@ class NetworkDisplay():
     def reinit_known_nodes(self):
         self.known_nodes_display = KnownNodes(self.app)
         self.known_nodes_display.delegate = self
-        self.close_list_dialogs()
         self.announce_stream_display.rebuild_widget_list()
 
     def reinit_lxmf_peers(self):
@@ -1676,6 +1810,12 @@ class NetworkDisplay():
         self.network_stats_display.start()
 
     def shortcuts(self):
+        try: focus_path = self.columns.get_focus_path()
+        except Exception: return self.list_shortcuts
+        try:
+            if focus_path[0] == 0: return self.list_shortcuts
+            if focus_path[0] == 1: return self.browser_shortcuts
+        except: pass
         return self.shortcuts_display
 
     def directory_change_callback(self):
@@ -1876,13 +2016,14 @@ def pretty_date(time=False):
     from datetime import datetime
     now = datetime.now()
     if type(time) is int:
-        diff = now - datetime.fromtimestamp(time)
+        then = datetime.fromtimestamp(time)
     elif isinstance(time,datetime):
-        diff = now - time
+        then = time
     elif not time:
-        diff = now - now
+        then = now
+    diff = now - then
     second_diff = diff.seconds
-    day_diff = diff.days
+    day_diff = (now.date() - then.date()).days
 
     if day_diff < 0:
         return ''

nomadnet/ui/textui/ReadlineEdit.py

@@ -0,0 +1,150 @@
+import urwid
+
+
+_KILL_KEYS = frozenset(("ctrl u", "ctrl k", "ctrl w", "ctrl l"))
+
+
+class _KillRing:
+    """Module-global kill buffer shared across all ReadlineMixin widgets.
+
+    Mirrors GNU readline: consecutive kills accumulate into the same entry
+    (forward kills append, backward kills prepend); any non-kill keypress
+    breaks the chain so the next kill replaces the buffer.
+    """
+    text = ""
+    last_was_kill = False
+
+    @classmethod
+    def reset_chain(cls):
+        cls.last_was_kill = False
+
+    @classmethod
+    def kill(cls, killed, direction):
+        if not killed:
+            return
+        if cls.last_was_kill:
+            cls.text = cls.text + killed if direction == "forward" else killed + cls.text
+        else:
+            cls.text = killed
+        cls.last_was_kill = True
+
+
+class ReadlineMixin:
+    """Mixin adding readline-style editing keys to an urwid.Edit-derived widget.
+
+    Bindings (GNU readline defaults, plus ctrl-l as kill-whole-buffer):
+        ctrl-a        beginning-of-line
+        ctrl-e        end-of-line
+        ctrl-u        unix-line-discard      (kill from cursor to beginning of line)
+        ctrl-k        kill-line              (kill from cursor to end of line)
+        ctrl-w        unix-word-rubout       (kill previous whitespace-delimited word)
+        ctrl-l        kill-whole-buffer      (kill the entire edit buffer)
+        ctrl-y        yank                   (insert most-recently-killed text)
+        ctrl-left     backward-word          (alphanumeric boundary)
+        ctrl-right    forward-word           (alphanumeric boundary)
+
+    "Line" is the current logical line within the edit buffer, delimited by
+    newlines -- so on multiline Edits these act on the line under the cursor,
+    not the entire buffer.
+
+    The kill buffer is shared across all widgets using this mixin, so text
+    killed in one Edit can be yanked into another.
+    """
+
+    def keypress(self, size, key):
+        if   key == "ctrl a":     self._rl_beg_of_line()
+        elif key == "ctrl e":     self._rl_end_of_line()
+        elif key == "ctrl u":     self._rl_kill_to_beg()
+        elif key == "ctrl k":     self._rl_kill_to_end()
+        elif key == "ctrl w":     self._rl_kill_word_back()
+        elif key == "ctrl l":     self._rl_kill_whole_buffer()
+        elif key == "ctrl y":     self._rl_yank()
+        elif key == "ctrl left":  self._rl_backward_word()
+        elif key == "ctrl right": self._rl_forward_word()
+        else:
+            result = super().keypress(size, key)
+            if key not in _KILL_KEYS:
+                _KillRing.reset_chain()
+            return result
+        if key not in _KILL_KEYS:
+            _KillRing.reset_chain()
+        return None
+
+    def _rl_line_bounds(self):
+        text, pos = self.edit_text, self.edit_pos
+        bol = text.rfind("\n", 0, pos)
+        bol = 0 if bol == -1 else bol + 1
+        eol = text.find("\n", pos)
+        eol = len(text) if eol == -1 else eol
+        return bol, eol
+
+    def _rl_delete(self, start, end, kill_direction=None):
+        if start == end:
+            return
+        text = self.edit_text
+        if kill_direction is not None:
+            _KillRing.kill(text[start:end], kill_direction)
+        self.set_edit_text(text[:start] + text[end:])
+        self.set_edit_pos(start)
+
+    @staticmethod
+    def _rl_is_word_char(ch):
+        return ch.isalnum() or ch == "_"
+
+    def _rl_beg_of_line(self):
+        bol, _ = self._rl_line_bounds()
+        self.set_edit_pos(bol)
+
+    def _rl_end_of_line(self):
+        _, eol = self._rl_line_bounds()
+        self.set_edit_pos(eol)
+
+    def _rl_kill_to_beg(self):
+        bol, _ = self._rl_line_bounds()
+        self._rl_delete(bol, self.edit_pos, kill_direction="backward")
+
+    def _rl_kill_to_end(self):
+        _, eol = self._rl_line_bounds()
+        self._rl_delete(self.edit_pos, eol, kill_direction="forward")
+
+    def _rl_kill_word_back(self):
+        text, pos = self.edit_text, self.edit_pos
+        p = pos
+        while p > 0 and text[p - 1].isspace():
+            p -= 1
+        while p > 0 and not text[p - 1].isspace():
+            p -= 1
+        self._rl_delete(p, pos, kill_direction="backward")
+
+    def _rl_kill_whole_buffer(self):
+        self._rl_delete(0, len(self.edit_text), kill_direction="forward")
+
+    def _rl_yank(self):
+        if not _KillRing.text:
+            return
+        pos = self.edit_pos
+        text = self.edit_text
+        self.set_edit_text(text[:pos] + _KillRing.text + text[pos:])
+        self.set_edit_pos(pos + len(_KillRing.text))
+
+    def _rl_backward_word(self):
+        text, pos = self.edit_text, self.edit_pos
+        while pos > 0 and not self._rl_is_word_char(text[pos - 1]):
+            pos -= 1
+        while pos > 0 and self._rl_is_word_char(text[pos - 1]):
+            pos -= 1
+        self.set_edit_pos(pos)
+
+    def _rl_forward_word(self):
+        text, pos = self.edit_text, self.edit_pos
+        n = len(text)
+        while pos < n and not self._rl_is_word_char(text[pos]):
+            pos += 1
+        while pos < n and self._rl_is_word_char(text[pos]):
+            pos += 1
+        self.set_edit_pos(pos)
+
+
+class ReadlineEdit(ReadlineMixin, urwid.Edit):
+    """Drop-in urwid.Edit replacement with readline-style editing keys."""
+    pass

nomadnet/ui/textui/Transport.py

@@ -0,0 +1,495 @@
+import RNS
+import time
+import urwid
+
+def _fmt_secs(secs):
+    secs = int(secs)
+    if secs < 0:
+        secs = 0
+    if secs < 60:
+        return "%ds" % secs
+    if secs < 3600:
+        return "%dm" % (secs // 60)
+    if secs < 86400:
+        return "%dh" % (secs // 3600)
+    return "%dd" % (secs // 86400)
+
+def _rel_future(ts):
+    if not ts:
+        return "-"
+    remaining = ts - time.time()
+    if remaining <= 0:
+        return "expired"
+    return _fmt_secs(remaining)
+
+def _rel_past(ts):
+    if not ts:
+        return "-"
+    return _fmt_secs(time.time() - ts) + " ago"
+
+def _fmt_duration(secs):
+    if not secs:
+        return "n/a"
+    secs = int(secs)
+    if secs < 0:
+        secs = 0
+    days = secs // 86400
+    hours = (secs % 86400) // 3600
+    minutes = (secs % 3600) // 60
+    if days:
+        return "%dd %dh" % (days, hours)
+    if hours:
+        return "%dh %dm" % (hours, minutes)
+    if minutes:
+        return "%dm" % minutes
+    return "%ds" % secs
+
+def _fmt_rate(hz):
+    try:
+        hz = float(hz or 0)
+    except Exception:
+        return "0"
+    if hz <= 0:
+        return "0"
+    per_hour = hz * 3600
+    if per_hour >= 1:
+        return "%.0f/h" % per_hour
+    return "%.1f/d" % (hz * 86400)
+
+def _mode_str(mode):
+    return {1: "Full", 2: "Point-to-Point", 3: "Access Point", 4: "Roaming", 5: "Boundary", 6: "Gateway"}.get(mode, "Full")
+
+class TransportRow(urwid.WidgetWrap):
+    def __init__(self, columns):
+        self._selectable = True
+        super().__init__(urwid.AttrMap(columns, "list_normal", focus_map="list_focus"))
+
+    def selectable(self):
+        return True
+
+    def keypress(self, size, key):
+        return key
+
+class LazyTableWalker(urwid.ListWalker):
+    def __init__(self):
+        self.items = []
+        self.factory = None
+        self.empty_widget = None
+        self.focus = 0
+
+    def configure(self, items, factory, empty_widget=None):
+        self.items = items or []
+        self.factory = factory
+        self.empty_widget = empty_widget
+        if self.focus >= len(self.items):
+            self.focus = max(0, len(self.items) - 1)
+        self._modified()
+
+    def _build(self, position):
+        return self.factory(self.items[position], position)
+
+    def get_focus(self):
+        if not self.items:
+            if self.empty_widget is not None:
+                return (self.empty_widget, 0)
+            return (None, None)
+        if self.focus >= len(self.items):
+            self.focus = len(self.items) - 1
+        if self.focus < 0:
+            self.focus = 0
+        return (self._build(self.focus), self.focus)
+
+    def set_focus(self, position):
+        self.focus = position
+        self._modified()
+
+    def get_next(self, position):
+        n = position + 1
+        if not self.items or n >= len(self.items):
+            return (None, None)
+        return (self._build(n), n)
+
+    def get_prev(self, position):
+        p = position - 1
+        if not self.items or p < 0:
+            return (None, None)
+        return (self._build(p), p)
+
+    def positions(self, reverse=False):
+        count = len(self.items)
+        if count == 0:
+            return [0] if self.empty_widget is not None else []
+        if reverse:
+            return range(count - 1, -1, -1)
+        return range(count)
+
+    def __len__(self):
+        return len(self.items)
+
+    def __getitem__(self, position):
+        if not self.items:
+            if position == 0 and self.empty_widget is not None:
+                return self.empty_widget
+            raise IndexError(position)
+        if position < 0 or position >= len(self.items):
+            raise IndexError(position)
+        return self._build(position)
+
+class TransportFiller(urwid.WidgetWrap):
+    def __init__(self, widget, display):
+        self.display = display
+        super().__init__(widget)
+
+    def keypress(self, size, key):
+        if key == "tab":
+            self.display.cycle_tab()
+            return None
+        if key == "ctrl r" or key == "f5":
+            self.display.refresh()
+            return None
+        if key == "esc":
+            self.display.go_back()
+            return None
+        result = super().keypress(size, key)
+        if key == "up" and result == "up":
+            self.display.go_back_to_header()
+            return None
+        return result
+
+class TransportDisplayShortcuts:
+    def __init__(self, app):
+        self.app = app
+        self.widget = urwid.AttrMap(
+            urwid.Text("[Tab] Switch Table  [Up] Filter  [C-r] Refresh  [Esc] Back"),
+            "shortcutbar"
+        )
+
+class TransportDisplay:
+    POLL_INTERVAL = 5
+
+    HASH_W = 34
+    HOPS_W = 6
+    VIA_W = 34
+    EXP_W = 12
+    LAST_W = 16
+    VIOL_W = 11
+    STATUS_W = 10
+    PATHS_W = 8
+    HELD_W = 8
+    QUEUE_W = 8
+    RATE_W = 10
+    MODE_W = 12
+
+    def __init__(self, app, parent=None):
+        self.app = app
+        self.parent = parent
+        self.started = False
+        self.poll_scheduler = False
+        self.glyphset = self.app.config["textui"]["glyphs"]
+        self.g = self.app.ui.glyphs
+
+        self.active_tab = "paths"
+        self.paths = []
+        self.rates = []
+        self.interfaces = []
+        self.paths_per_interface = {}
+        self.link_count = 0
+        self.transport_id = None
+        self.network_id = None
+        self.transport_uptime = None
+        self.held_announces = 0
+        self.filter_text = ""
+
+        self.summary_text = urwid.Text("")
+        self.list_walker = LazyTableWalker()
+        self.list_box = urwid.ListBox(self.list_walker)
+
+        self.filter_edit = urwid.Edit("", "")
+        urwid.connect_signal(self.filter_edit, "change", self._on_filter_change)
+        self.filter_row = urwid.Columns([
+            (9, urwid.Text(("key", "Filter:"))),
+            urwid.AttrMap(self.filter_edit, "list_normal", focus_map="list_focus"),
+        ], dividechars=1)
+
+        self.pile = urwid.Pile([
+            ('pack', self.summary_text),
+            ('pack', self._tab_bar()),
+            ('pack', self.filter_row),
+            ('pack', self._header_row()),
+            ('pack', urwid.Divider("─")),
+            ('weight', 1, self.list_box),
+        ])
+
+        self.transport_view = TransportFiller(self.pile, self)
+        self.shortcuts_display = TransportDisplayShortcuts(self.app)
+        self.widget = self.transport_view
+
+        self._refresh_data()
+        self._rebuild()
+
+    def shortcuts(self):
+        return self.shortcuts_display
+
+    def _tab_bar(self):
+        def tab(label, key):
+            attr = "list_focus" if self.active_tab == key else "list_normal"
+            return (urwid.PACK, urwid.Text((attr, " %s " % label)))
+
+        return urwid.Columns([
+            tab("Paths", "paths"),
+            tab("Announce Rates", "rates"),
+            tab("Interfaces", "interfaces"),
+            (urwid.WEIGHT, 1, urwid.Text("")),
+        ], dividechars=1)
+
+    def _header_row(self):
+        if self.active_tab == "paths":
+            cols = [
+                (self.HASH_W, urwid.Text(("key", "Destination"), wrap="clip")),
+                (self.HOPS_W, urwid.Text(("key", "Hops"), wrap="clip")),
+                (urwid.WEIGHT, 1, urwid.Text(("key", "Interface"), wrap="clip")),
+                (self.VIA_W, urwid.Text(("key", "Via"), wrap="clip")),
+                (self.EXP_W, urwid.Text(("key", "Expires"), wrap="clip")),
+            ]
+        elif self.active_tab == "interfaces":
+            cols = [
+                (urwid.WEIGHT, 1, urwid.Text(("key", "Interface"), wrap="clip")),
+                (self.STATUS_W, urwid.Text(("key", "Status"), wrap="clip")),
+                (self.PATHS_W, urwid.Text(("key", "Paths"), wrap="clip")),
+                (self.HELD_W, urwid.Text(("key", "Held"), wrap="clip")),
+                (self.QUEUE_W, urwid.Text(("key", "Queue"), wrap="clip")),
+                (self.RATE_W, urwid.Text(("key", "Ann in"), wrap="clip")),
+                (self.MODE_W, urwid.Text(("key", "Mode"), wrap="clip")),
+            ]
+        else:
+            cols = [
+                (self.HASH_W, urwid.Text(("key", "Destination"), wrap="clip")),
+                (self.LAST_W, urwid.Text(("key", "Last announce"), wrap="clip")),
+                (self.VIOL_W, urwid.Text(("key", "Violations"), wrap="clip")),
+                (urwid.WEIGHT, 1, urwid.Text(("key", "Blocked"), wrap="clip")),
+            ]
+        return urwid.Columns(cols, dividechars=1)
+
+    def _refresh_data(self):
+        try:
+            self.paths = self.app.rns.get_path_table() or []
+        except Exception:
+            self.paths = []
+        try:
+            self.rates = self.app.rns.get_rate_table() or []
+        except Exception:
+            self.rates = []
+        try:
+            self.link_count = self.app.rns.get_link_count()
+        except Exception:
+            self.link_count = 0
+
+        try:
+            stats = self.app.rns.get_interface_stats() or {}
+        except Exception:
+            stats = {}
+        self.transport_id = stats.get("transport_id")
+        self.network_id = stats.get("network_id")
+        self.transport_uptime = stats.get("transport_uptime")
+        self.interfaces = stats.get("interfaces", []) or []
+        try:
+            self.held_announces = sum(int(i.get("held_announces", 0) or 0) for i in self.interfaces)
+        except Exception:
+            self.held_announces = 0
+
+        self.paths_per_interface = {}
+        for entry in self.paths:
+            iface = entry.get("interface", "")
+            self.paths_per_interface[iface] = self.paths_per_interface.get(iface, 0) + 1
+
+        try:
+            self.interfaces.sort(key=lambda i: self.paths_per_interface.get(i.get("name", ""), 0), reverse=True)
+        except Exception:
+            pass
+
+        try:
+            self.paths.sort(key=lambda e: (e.get("interface", ""), e.get("hops", 0)))
+        except Exception:
+            pass
+        try:
+            self.rates.sort(key=lambda e: e.get("last", 0), reverse=True)
+        except Exception:
+            pass
+
+    def _make_path_row(self, entry, position):
+        dest = RNS.prettyhexrep(entry["hash"]) if entry.get("hash") else "?"
+        via = RNS.prettyhexrep(entry["via"]) if entry.get("via") else "?"
+        hops = entry.get("hops", 0)
+        iface = entry.get("interface", "")
+        expires = _rel_future(entry.get("expires", 0))
+
+        columns = urwid.Columns([
+            (self.HASH_W, urwid.Text(("list_normal", dest), wrap="clip")),
+            (self.HOPS_W, urwid.Text(str(hops), wrap="clip")),
+            (urwid.WEIGHT, 1, urwid.Text(iface, wrap="clip")),
+            (self.VIA_W, urwid.Text(via, wrap="clip")),
+            (self.EXP_W, urwid.Text(expires, wrap="clip")),
+        ], dividechars=1)
+        return TransportRow(columns)
+
+    def _make_rate_row(self, entry, position):
+        now = time.time()
+        dest = RNS.prettyhexrep(entry["hash"]) if entry.get("hash") else "?"
+        last = _rel_past(entry.get("last", 0))
+        violations = str(entry.get("rate_violations", 0))
+        blocked_until = entry.get("blocked_until", 0) or 0
+        if blocked_until > now:
+            blocked = ("disconnected_status", "blocked %s" % _rel_future(blocked_until))
+        else:
+            blocked = ("connected_status", "no")
+
+        columns = urwid.Columns([
+            (self.HASH_W, urwid.Text(("list_normal", dest), wrap="clip")),
+            (self.LAST_W, urwid.Text(last, wrap="clip")),
+            (self.VIOL_W, urwid.Text(violations, wrap="clip")),
+            (urwid.WEIGHT, 1, urwid.Text(blocked, wrap="clip")),
+        ], dividechars=1)
+        return TransportRow(columns)
+
+    def _make_iface_row(self, entry, position):
+        short_name = entry.get("short_name")
+        if short_name == "None": short_name = None
+        name = short_name or entry.get("name") or "?"
+        RNS.log(name)
+        name = name.replace("\x00", "")
+        paths = self.paths_per_interface.get(entry.get("name", ""), 0)
+        held = entry.get("held_announces", 0) or 0
+        queue = entry.get("announce_queue")
+        queue_str = "-" if queue is None else str(queue)
+        rate = _fmt_rate(entry.get("incoming_announce_frequency", 0))
+        mode = _mode_str(entry.get("mode"))
+
+        if entry.get("status"):
+            status = ("connected_status", "up")
+        else:
+            status = ("disconnected_status", "down")
+
+        columns = urwid.Columns([
+            (urwid.WEIGHT, 1, urwid.Text(("list_normal", name), wrap="clip")),
+            (self.STATUS_W, urwid.Text(status, wrap="clip")),
+            (self.PATHS_W, urwid.Text(str(paths), wrap="clip")),
+            (self.HELD_W, urwid.Text(str(held), wrap="clip")),
+            (self.QUEUE_W, urwid.Text(queue_str, wrap="clip")),
+            (self.RATE_W, urwid.Text(rate, wrap="clip")),
+            (self.MODE_W, urwid.Text(mode, wrap="clip")),
+        ], dividechars=1)
+        return TransportRow(columns)
+
+    def _rebuild(self):
+        blocked_now = 0
+        try:
+            now = time.time()
+            blocked_now = sum(1 for r in self.rates if (r.get("blocked_until", 0) or 0) > now)
+        except Exception:
+            pass
+
+        transport_id = RNS.prettyhexrep(self.transport_id) if self.transport_id else "n/a"
+        network_id = RNS.prettyhexrep(self.network_id) if self.network_id else "n/a"
+        uptime = _fmt_duration(self.transport_uptime)
+
+        summary = [
+            ("key", "Transport ID: "), ("value", transport_id),
+            ("key", "   Network ID: "), ("value", network_id),
+            ("key", "   Uptime: "), ("value", uptime), "\n",
+            ("key", "Active links: "), ("value", str(self.link_count)),
+            ("key", "   Known paths: "), ("value", str(len(self.paths))),
+            ("key", "   Held announces: "), ("value", str(self.held_announces)),
+            ("key", "   Announce sources: "), ("value", str(len(self.rates))),
+            ("key", "   Rate-limited: "), ("value", str(blocked_now)),
+        ]
+
+        if self.active_tab == "paths":
+            data = self._filtered(self.paths)
+            factory = self._make_path_row
+            empty_msg = "No paths in the routing table yet."
+        elif self.active_tab == "interfaces":
+            data = self._filtered(self.interfaces)
+            factory = self._make_iface_row
+            empty_msg = "No interface statistics available."
+        else:
+            data = self._filtered(self.rates)
+            factory = self._make_rate_row
+            empty_msg = "No announce rate data available."
+
+        if self.filter_text.strip():
+            summary.append(("inactive_text", "   [filter: %d shown]" % len(data)))
+        self.summary_text.set_text(summary)
+
+        self.pile.contents[1] = (self._tab_bar(), self.pile.options('pack'))
+        self.pile.contents[3] = (self._header_row(), self.pile.options('pack'))
+
+        empty = urwid.Text(("inactive_text", empty_msg), align=urwid.CENTER)
+        self.list_walker.configure(data, factory, empty)
+
+    def cycle_tab(self):
+        order = ["paths", "rates", "interfaces"]
+        i = order.index(self.active_tab) if self.active_tab in order else 0
+        self.active_tab = order[(i + 1) % len(order)]
+        self._rebuild()
+
+    def _on_filter_change(self, widget, new_text):
+        self.filter_text = new_text
+        self._rebuild()
+
+    def _haystack(self, entry):
+        parts = []
+        for key in ("hash", "via"):
+            value = entry.get(key)
+            if value:
+                parts.append(RNS.prettyhexrep(value))
+        for key in ("interface", "name", "short_name"):
+            value = entry.get(key)
+            if value:
+                parts.append(str(value))
+        return " ".join(parts).lower()
+
+    def _filtered(self, items):
+        needle = self.filter_text.strip().lower()
+        if not needle:
+            return items
+        return [entry for entry in items if needle in self._haystack(entry)]
+
+    def go_back(self):
+        if self.parent is not None:
+            self.parent.show_landing()
+
+    def go_back_to_header(self):
+        try:
+            self.app.ui.main_display.frame.focus_position = "header"
+        except Exception:
+            pass
+
+    def _is_visible(self):
+        md = self.app.ui.main_display
+        if self.parent is not None:
+            return md.sub_displays.active_display is self.parent and getattr(self.parent, "active", None) == "transport"
+        return md.sub_displays.active_display is self
+
+    def refresh(self):
+        self._refresh_data()
+        self._rebuild()
+
+    def start(self):
+        self.started = True
+        self._refresh_data()
+        self._rebuild()
+        if not self.poll_scheduler and self.app.ui.loop is not None:
+            self.poll_scheduler = True
+            self.app.ui.loop.set_alarm_in(self.POLL_INTERVAL, self._poll)
+
+    def _poll(self, loop, user_data):
+        if not self.started or not self._is_visible():
+            self.poll_scheduler = False
+            return
+        self._refresh_data()
+        self._rebuild()
+        try:
+            loop.draw_screen()
+        except Exception:
+            pass
+        loop.set_alarm_in(self.POLL_INTERVAL, self._poll)

nomadnet/ui/textui/images/__init__.py

@@ -0,0 +1,18 @@
+# A minimal, dependency-free image widget for urwid, rendering images through
+# the Kitty terminal graphics protocol.
+
+from ._termlib import get_cell_size, is_kitty_supported, reset_caches, diagnostics
+from ._imagedata import ImageData, parse_image_header
+from .widget import ImageWidget, ImageCanvas, UI_MARGIN
+from .screen import ImageScreen
+
+__all__ = ( "ImageWidget",
+            "ImageCanvas",
+            "ImageScreen",
+            "ImageData",
+            "parse_image_header",
+            "is_kitty_supported",
+            "get_cell_size",
+            "diagnostics",
+            "reset_caches",
+            "UI_MARGIN" )

nomadnet/ui/textui/images/_ctlseqs.py

@@ -0,0 +1,78 @@
+# Kitty Terminal Graphics Protocol control sequences
+# 
+# Minimal set of control sequences needed for the Kitty graphics protocol
+# side-channel image rendering used by the urwid image widget.
+
+from base64 import standard_b64encode
+import re
+
+# Basic building blocks
+APC = b"\x1b_" # Application Program Command
+ST = b"\x1b\\" # String Terminator
+CSI = b"\x1b["
+
+# Kitty graphics: transmit-and-display with chunking support.
+#   ESC _ G <control-data> ; <base64 payload> ESC \
+KITTY_TRANSMISSION = b"\x1b_G%s;%s\x1b\\"
+
+# Delete images: d=A (all placements visible on screen), d=i (one
+# placement, data kept), d=I (whole image, data freed).
+KITTY_DELETE_ALL = b"\x1b_Ga=d,d=A\x1b\\"
+KITTY_DELETE_PLACEMENT = b"\x1b_Ga=d,d=i,i=%d,p=%d\x1b\\"
+KITTY_DELETE_IMAGE = b"\x1b_Ga=d,d=I,i=%d\x1b\\"
+
+# Support query (payload is base64 of 4 zero bytes; expects "i=31;OK")
+KITTY_QUERY = b"\x1b_Ga=q,t=d,i=31,f=24,s=1,v=1,C=1,c=1,r=1;AAAA\x1b\\"
+KITTY_QUERY_RESPONSE_RE = re.compile(b"\x1b_Gi=31;(?P<message>[^\x1b]*)\x1b\\\\")
+
+# Cursor: absolute positioning, erase characters, cursor forward
+CURSOR_POSITION = b"\x1b[%d;%dH"
+ERASE_CHARS = b"\x1b[%dX"
+
+# Synchronized output (DEC 2026) - reduces flicker/tearing when redrawing
+BEGIN_SYNCED_UPDATE = b"\x1b[?2026h"
+END_SYNCED_UPDATE = b"\x1b[?2026l"
+
+# Terminal queries: cell size in pixels (XTWINOPS 16) + primary DA
+CELL_SIZE_PX = b"\x1b[16t"
+DA1 = b"\x1b[c"
+# Response: CSI 4 ; <height-px> ; <width-px> t
+CELL_SIZE_RESPONSE_RE = re.compile(b"\x1b\\[4;(?P<h>\\d+);(?P<w>\\d+)t")
+
+# Maximum payload (base64) per transmission chunk, as recommended by the
+# protocol documentation
+CHUNK_SIZE = 4096
+
+# Control data for a transmit-only (a=t) with a quiet mode (q=1)
+def kitty_transmit_control(image_id): return b"a=t,i=%d,f=100,t=d,q=1" % image_id
+
+# A placement command (a=p) for a previously transmitted image.
+# Re-sending the same (image id, placement id) pair replaces the
+# placement in place - used to move/resize/slice without flicker.
+# y/h select the source rectangle (image pixels) when a slice of
+# the image is displayed.
+def kitty_placement(image_id, placement_id, cols, rows, z, y=None, h=None):
+    control = b"a=p,i=%d,p=%d,c=%d,r=%d,z=%d,C=1,q=1" % (image_id, placement_id, cols, rows, z)
+    if y is not None and h is not None: control += b",y=%d,h=%d" % (y, h)
+    return KITTY_TRANSMISSION % (control, b"")
+
+# Encodes a kitty graphics transmission, chunking the base64-encoded
+# payload into ``chunk_size``-byte pieces.
+def kitty_transmission(control, payload, chunk_size=CHUNK_SIZE):
+    encoded = standard_b64encode(payload)
+    chunks = [encoded[i:i + chunk_size] for i in range(0, len(encoded), chunk_size)]
+    if not chunks:
+        chunks = [b""]
+
+    def control_for(index, n_chunks):
+        if n_chunks == 1:
+            return control
+        if index == 0:
+            # The control data appears only in the first chunk; the chunk
+            # marker says more chunks are coming.
+            return control + b",m=1"
+        if index == n_chunks - 1:
+            return b"m=0"
+        return b"m=1"
+
+    return b"".join(KITTY_TRANSMISSION % (control_for(i, len(chunks)), chunk) for i, chunk in enumerate(chunks))

nomadnet/ui/textui/images/_imagedata.py

@@ -0,0 +1,142 @@
+# Image file handling for the side-channel renderer
+#
+# The Kitty graphics protocol can decode and scale images in the terminal
+# itself (data format ``f=100``), which means the widget needs no image
+# decoding library at all. It only needs the *dimensions* of the image to lay
+# it out correctly in the urwid canvas; those are trivially parseable from the
+# file headers of the supported formats.
+# 
+# Supported source formats:
+#  - WebP (VP8, VP8L and VP8X)
+#  - PNG
+# 
+# The payload transmitted to the terminal is always a PNG file: PNG sources
+# are passed through verbatim, and WebP sources are converted transparently
+# by the multi-backend conversion chain in :mod:`._webp` (no required
+# dependencies). The conversion result is cached on disk, so each unique
+# WebP is converted at most once per system.
+
+import hashlib
+import os
+
+MAX_PAYLOAD_BYTES           = 8 * 1024 * 1024
+MAX_CONVERTED_PAYLOAD_BYTES = 16 * 1024 * 1024
+_PNG_SIGNATURE              = b"\x89PNG\r\n\x1a\n"
+
+# Returns (width, height) from a PNG file.
+def _parse_png(data):
+    if not data.startswith(_PNG_SIGNATURE): raise ValueError("not a PNG")
+    if len(data) < 24 or data[12:16] != b"IHDR": raise ValueError("PNG missing IHDR chunk")
+    width = int.from_bytes(data[16:20], "big")
+    height = int.from_bytes(data[20:24], "big")
+    if not (width > 0 and height > 0): raise ValueError("PNG has null dimensions")
+    return width, height
+
+# Returns (width, height) from a WebP file (VP8, VP8L or VP8X).
+def _parse_webp(data):
+    if not data.startswith(b"RIFF") or data[8:12] != b"WEBP": raise ValueError("not a WebP")
+    if len(data) < 30: raise ValueError("WebP file too short")
+    fourcc = data[12:16]
+    if fourcc == b"VP8X":
+        # 10-byte chunk: flags(1) reserved(3) canvas-w-1(3 LE) canvas-h-1(3 LE)
+        width = int.from_bytes(data[24:27], "little") + 1
+        height = int.from_bytes(data[27:30], "little") + 1
+    elif fourcc == b"VP8 ":
+        # frame tag(3) start code 0x9d012a(3) width(2 LE, 14 bits) height(2 LE, 14 bits)
+        width = int.from_bytes(data[26:28], "little") & 0x3FFF
+        height = int.from_bytes(data[28:30], "little") & 0x3FFF
+    elif fourcc == b"VP8L":
+        # signature(1) then 4 bytes: w-1 (14 bits) | h-1 (14 bits) | ...
+        bits = int.from_bytes(data[21:25], "little")
+        width = (bits & 0x3FFF) + 1
+        height = ((bits >> 14) & 0x3FFF) + 1
+    else: raise ValueError("unsupported WebP variant %r" % fourcc)
+    if not (width > 0 and height > 0): raise ValueError("WebP has null dimensions")
+    return width, height
+
+
+def parse_image_header(data):
+    if data[:8] == _PNG_SIGNATURE:
+        w, h = _parse_png(data)
+        return "PNG", w, h
+    if data[:4] == b"RIFF" and data[8:12] == b"WEBP":
+        w, h = _parse_webp(data)
+        return "WebP", w, h
+    raise ValueError("unsupported image format")
+
+# Loads an image file and exposes its PNG bytes and dimensions.
+# The payload is always a PNG file, transmitted verbatim to the terminal
+# (Kitty's f=100 data format). The dimensions are used only for layout.
+# WebP sources are converted transparently to PNG on load.
+class ImageData(object):
+    def __init__(self, path, max_bytes=MAX_PAYLOAD_BYTES, remote_source=False):
+        self.path = path
+        self.error = None
+        self.format = None
+        self.source_format = None
+        self.conversion_backend = None
+        self.width = 0
+        self.height = 0
+        self.data = None
+        try:
+            with open(path, "rb") as f: data = f.read()
+        except OSError as e:
+            self.error = "Could not read file: %s" % e
+            return
+        
+        if len(data) > max_bytes:
+            self.error = "File too large (%.1f MiB > %d MiB)" % (len(data) / (1024 * 1024), max_bytes // (1024 * 1024))
+            return
+        
+        if not data:
+            self.error = "Empty file"
+            return
+        
+        try: self.format, self.width, self.height = parse_image_header(data)
+        except ValueError as e:
+            self.error = str(e)
+            return
+        
+        self.source_format = self.format
+        if self.format == "WebP":
+            from . import _webp
+            png = _webp.convert_webp_to_png(data)
+            if png is None:
+                self.error = "WebP conversion failed (no working backend available)"
+                return
+            if len(png) > MAX_CONVERTED_PAYLOAD_BYTES:
+                self.error = "Converted image too large (%.1f MiB > %d MiB)" % (len(png) / (1024 * 1024), MAX_CONVERTED_PAYLOAD_BYTES // (1024 * 1024))
+                return
+
+            self.data = png
+            self.format = "PNG"
+            self.conversion_backend = _webp.last_backend()
+        
+        else:
+            if remote_source:
+                self.error = "Invalid image format, remote images must be WebP"
+                return
+            else:
+                self.data = data
+        
+        # Content key for de-duplication: identical bytes = same key, so
+        # the same image transmitted once can be placed many times.
+        self._key = hashlib.sha256(self.data).hexdigest()
+
+    @property
+    def ok(self): return self.data is not None
+
+    @property
+    def key(self): return getattr(self, "_key", None)
+
+    @property
+    def size(self): return (self.width, self.height)
+
+    def description(self):
+        if self.ok:
+            if self.source_format == "WebP":
+                return "WebP %dx%d -> PNG (%s)" % (self.width, self.height, self.conversion_backend or "cached")
+            else:
+                return "%s %dx%d" % (self.format, self.width, self.height)
+
+        return self.error

nomadnet/ui/textui/images/_imagestore.py

@@ -0,0 +1,90 @@
+# Registry of terminal-side image data
+# 
+# Tracks which image payloads have been transmitted to the terminal (Kitty
+# image ids), de-duplicates identical image data across widgets (one payload
+# transmitted once, placed many times), and counts live widget references so
+# that orphaned data can be evicted from the terminal's buffers.
+# 
+# This module only tracks *state*; all protocol output is performed by
+# ImageScreen, which reads and updates the store.
+# 
+# Eviction:
+# 
+# - Automatic, widgets unregister when closed/destroyed; the screen evicts
+#   (deletes from the terminal) any registered data with no live references.
+# - Explicit, the browser can call ImageScreen.purge_images() on page
+#   navigation to drop everything at once.
+
+DEFAULT_IMAGE_ID_MAX = 2 ** 32 - 1  # Protocol limit for image ids
+
+class ImageStore(object):
+    def __init__(self):
+        self._entries = {}  # content key -> {"image_id", "data", "refs", "transmitted"}
+        self._next_image_id = 1
+
+    #############
+    # Lifecycle #
+    #############
+
+    # Registers image data and returns (key, is_new).
+    # If identical data is already registered, the existing entry is
+    # reused (de-duplication) and its reference count is incremented.
+    def register(self, key, data):
+        entry = self._entries.get(key)
+        if entry is None:
+            image_id = self._next_image_id
+            self._next_image_id += 1
+            if image_id > DEFAULT_IMAGE_ID_MAX: raise OverflowError("image id space exhausted")
+            entry = { "image_id": image_id, "data": data,
+                      "refs": 0, "transmitted": False }
+            self._entries[key] = entry
+            is_new = True
+        else: is_new = False
+
+        entry["refs"] += 1
+        return key, is_new
+
+    # Decrements the reference count of registered data
+    def unregister(self, key):
+        entry = self._entries.get(key)
+        if entry is None: return
+        entry["refs"] = max(0, entry["refs"] - 1)
+
+    ###########
+    # Queries #
+    ###########
+
+    def entry_for(self, key): return self._entries.get(key)
+
+    def transmitted(self): return [key for key, entry in self._entries.items() if entry["transmitted"]]
+
+    # Returns [(key, image_id), ...] for entries with no live
+    # widget references that are still present in the terminal.
+    def evict_orphans(self):
+        return [ (key, entry["image_id"])
+                 for key, entry in self._entries.items()
+                 if entry["refs"] == 0 and entry["transmitted"] ]
+
+    ############
+    # Mutation #
+    ############
+
+    def mark_transmitted(self, key):
+        entry = self._entries.get(key)
+        if entry is not None:
+            entry["transmitted"] = True
+
+    def mark_all_untransmitted(self):
+        for entry in self._entries.values():
+            entry["transmitted"] = False
+
+    # Drops the entry for evicted data (frees the payload bytes)
+    def evict(self, key): self._entries.pop(key, None)
+
+    # Forgets all state (used by tests)
+    def reset(self):
+        self._entries.clear()
+        self._next_image_id = 1
+
+# Module-level singleton shared by widgets and the screen.
+image_store = ImageStore()

nomadnet/ui/textui/images/_selftest.py

@@ -0,0 +1,718 @@
+#!/usr/bin/env python3
+"""
+.. Images - headless self test
+
+Exercises the widget + screen machinery without a terminal: image header
+parsing, display sizing, shard scanning, APC construction/chunking and the
+transmit/delete state machine at various scroll positions.
+
+Run::
+
+    python3 -m nomadnet.ui.textui.images._selftest [image-file]
+"""
+
+import base64
+import io
+import os
+import shutil
+import struct
+import sys
+import tempfile
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", ".."))
+
+import urwid
+
+from nomadnet.ui.textui.images import _ctlseqs
+from nomadnet.ui.textui.images import _imagestore
+from nomadnet.ui.textui.images import _termlib
+from nomadnet.ui.textui.images._imagedata import ImageData, parse_image_header
+from nomadnet.ui.textui.images.screen import ImageScreen
+from nomadnet.ui.textui.images.widget import ImageWidget
+
+DEFAULT_IMAGE = os.path.expanduser("~/Scratchpad/test.png")
+VIEWPORT = (80, 24)
+CELL_SIZE = (1, 2)  # forced fallback for deterministic layout
+
+PASS = 0
+FAIL = 0
+
+
+def check(cond, label):
+    global PASS, FAIL
+    if cond:
+        PASS += 1
+        print("  ok   - %s" % label)
+    else:
+        FAIL += 1
+        print("  FAIL - %s" % label)
+
+
+def build_pile(image_path, wrap_attrmap=False):
+    widgets = []
+    for i in range(12):
+        widgets.append(urwid.Text("[%02d]\nline one\nline two\n" % i))  # 3 rows each
+    img = ImageWidget(image_path)
+    if wrap_attrmap:
+        img = urwid.AttrMap(img, "body")
+    widgets.append(urwid.Divider(" "))
+    widgets.append(img)
+    widgets.append(urwid.Divider(" "))
+    for i in range(12):
+        widgets.append(urwid.Text("[%02d]\nline one\nline two\n" % (13 + i)))
+    return urwid.Pile(widgets)
+
+
+class RecordingScreen(ImageScreen):
+    def __init__(self):
+        super().__init__()
+        self.log = []
+
+    def write(self, data):
+        if isinstance(data, bytes):
+            data = data.decode("ascii")
+        self.log.append(data)
+        return data
+
+    def flush(self):
+        pass
+
+
+def extract_transmit(log):
+    """Returns (image_id, joined_b64_payload) of the a=t transmission, or
+    None if no transmission is present."""
+    image_id = None
+    payloads = []
+    for item in log:
+        if item.startswith("\x1b_Ga=t"):
+            for piece in item.split("\x1b_G")[1:]:
+                body = piece[:-2]  # strip the trailing ST (ESC \)
+                head, sep, payload = body.partition(";")
+                if not sep:
+                    continue
+                if image_id is None:
+                    image_id = int(head.split("i=")[1].split(",")[0])
+                payloads.append(payload)
+    if image_id is None:
+        return None
+    return image_id, "".join(payloads)
+
+
+def extract_placements(log):
+    """Returns the control data of every a=p placement command."""
+    return [
+        item[3:-2]
+        for item in log
+        if item.startswith("\x1b_Ga=p")
+    ]
+
+
+def extract_deletes(log):
+    """Returns the control data of every a=d delete command."""
+    return [
+        item[3:-2]
+        for item in log
+        if item.startswith("\x1b_Ga=d")
+    ]
+
+
+def screen_probe_placement_key(widget, row, col, canvas_cols, trim_top, vis_rows):
+    """Computes an ImageScreen placement key without a screen instance."""
+    return ImageScreen._placement_key(
+        None, widget, row, col, canvas_cols, trim_top, vis_rows)
+
+
+def test_header_parse(image_path):
+    print("== header parsing ==")
+    with open(image_path, "rb") as f:
+        data = f.read()
+    fmt, w, h = parse_image_header(data)
+    check(fmt == "PNG", "detected PNG")
+    check(w == 1280 and h == 960, "dimensions %dx%d" % (w, h))
+    imgdata = ImageData(image_path)
+    check(imgdata.ok, "ImageData loads without error")
+    check(imgdata.size == (w, h), "ImageData size matches")
+
+
+def test_transmission_encoding(image_path):
+    print("== APC encoding/chunking ==")
+    small = b"\x89PNG\r\n\x1a\n" + b"x" * 100
+    single = _ctlseqs.kitty_transmission(b"a=T,f=100,c=80,r=30,z=1", small)
+    check(single.count(b"\x1b_G") == 1, "single-chunk transmission has one APC")
+    check(single.startswith(b"\x1b_Ga=T,f=100,c=80,r=30,z=1;"), "control data correct")
+    check(single.endswith(b"\x1b\\"), "ends with ST")
+    payload = single.split(b";", 1)[1][:-2]
+    check(base64.b64decode(payload) == small, "payload round-trips to original file")
+
+    with open(image_path, "rb") as f:
+        data = f.read()
+    chunks = _ctlseqs.kitty_transmission(b"a=T,f=100", data, chunk_size=16)
+    apcs = chunks.split(b"\x1b\\")[:-1]
+    check(len(apcs) > 1, "multi-chunk transmission splits into %d APCs" % len(apcs))
+    first, last = apcs[0], apcs[-1]
+    heads = [a.split(b";", 1)[0][3:] for a in apcs]
+    check(b",m=1;" in first, "first chunk carries control + m=1")
+    check(last.startswith(b"\x1b_Gm=0;"), "last chunk carries m=0")
+    check(sum(b"a=T" in h for h in heads) == 1, "control data appears only once")
+    check(all(h == b"m=1" or h == b"m=0" for h in heads[1:]), "continuation chunks carry only the chunk marker")
+    joined_b64 = b"".join(a.split(b";", 1)[1] for a in apcs)
+    check(base64.b64decode(joined_b64) == data, "multi-chunk payload reassembles")
+
+
+_TINY_WEBP = base64.b64decode(
+    "UklGRiQAAABXRUJQVlA4IBgAAAAwAQCdASoBAAEAAUAmJaQAA3AA/vuUAAA=")
+
+
+def _webp_fixture():
+    """Returns (webp_bytes, (width, height)): a 4x4 WebP generated with
+    Pillow when available (more representative), else a 1x1 constant."""
+    try:
+        from PIL import Image
+        if _webp_module_available():
+            image = Image.new("RGBA", (4, 4))
+            for y in range(4):
+                for x in range(4):
+                    image.putpixel((x, y), (x * 60, y * 60, x * y * 10, 255))
+            buf = io.BytesIO()
+            image.save(buf, format="WEBP")
+            data = buf.getvalue()
+            if parse_image_header(data)[0] == "WebP":
+                return data, (4, 4)
+    except Exception:
+        pass
+    return _TINY_WEBP, (1, 1)
+
+
+def _webp_module_available():
+    try:
+        from PIL import features
+        return bool(features.check("webp"))
+    except Exception:
+        return False
+
+
+def test_webp_conversion():
+    print("== webp conversion ==")
+    from nomadnet.ui.textui.images import _webp
+    from nomadnet.ui.textui.images._imagedata import MAX_CONVERTED_PAYLOAD_BYTES
+
+    fmt, w, h = parse_image_header(_TINY_WEBP)
+    check((fmt, w, h) == ("WebP", 1, 1), "embedded fixture is a 1x1 WebP")
+
+    fixture, dims = _webp_fixture()
+    check(parse_image_header(fixture)[1:] == dims,
+          "webp fixture has expected dimensions")
+
+    cache_tmp = tempfile.mkdtemp(prefix="imgw-test-cache-")
+    prev_cache_env = os.environ.get("NOMADNET_IMAGE_CACHE")
+    os.environ["NOMADNET_IMAGE_CACHE"] = cache_tmp
+    try:
+        # Each available backend must convert the fixture to a valid PNG
+        # with matching dimensions. Backends are invoked directly, since
+        # the on-disk cache would otherwise mask all but the first one.
+        _webp.reset()
+        present = [name for name, avail in _webp.available_backends() if avail]
+        converted = 0
+        for name in present:
+            png = _webp._BACKENDS[name](fixture, dims)
+            ok = (png is not None and png.startswith(b"\x89PNG")
+                  and _webp._png_dims(png) == dims)
+            check(ok, "%s backend converts to a valid PNG" % name)
+            if ok:
+                converted += 1
+        check(converted >= 1, "at least one backend converts webp (got %d)" % converted)
+
+        # Repeat conversion is served from the on-disk cache: the backend
+        # table is disabled and the conversion still succeeds.
+        _webp.reset()
+        png1 = _webp.convert_webp_to_png(fixture)
+        check(png1 is not None, "first conversion succeeds")
+        orig_backends = dict(_webp._BACKENDS)
+        _webp._BACKENDS = {name: (lambda *a, **k: None) for name in _webp.BACKENDS}
+        png2 = _webp.convert_webp_to_png(fixture)
+        check(png1 == png2 and _webp.stats()["cache_hits"] >= 1,
+              "repeat conversion served from the on-disk cache")
+        _webp._BACKENDS = orig_backends
+
+        # ImageData-level transparency: a webp file loads as a PNG.
+        _webp.reset()
+        fd, webp_path = tempfile.mkstemp(suffix=".webp")
+        with os.fdopen(fd, "wb") as f:
+            f.write(fixture)
+        d = ImageData(webp_path)
+        check(d.ok and d.format == "PNG" and d.size == dims,
+              "ImageData converts webp transparently to PNG")
+        check(d.data.startswith(b"\x89PNG\r\n\x1a\n") and d.key is not None,
+              "converted payload is a keyed PNG")
+        check("-> PNG" in d.description(), "description notes the conversion")
+        os.unlink(webp_path)
+
+        # Converted-payload cap.
+        _webp.reset()
+        orig_convert = _webp.convert_webp_to_png
+        _webp.convert_webp_to_png = lambda data: b"x" * (MAX_CONVERTED_PAYLOAD_BYTES + 1)
+        fd, webp_path = tempfile.mkstemp(suffix=".webp")
+        with os.fdopen(fd, "wb") as f:
+            f.write(fixture)
+        d = ImageData(webp_path)
+        check(not d.ok and "Converted image too large" in d.error,
+              "converted payload capped at %d MiB"
+              % (MAX_CONVERTED_PAYLOAD_BYTES // (1024 * 1024)))
+        os.unlink(webp_path)
+        _webp.convert_webp_to_png = orig_convert
+
+        # Undecodable webp (valid header/dimensions, garbage bitstream).
+        _webp.reset()
+        junk = (b"RIFF" + struct.pack("<I", 30) + b"WEBP" + b"VP8 "
+                + b"\x00" * 10 + b"\x10\x00\x10\x00" + b"\xff" * 12)
+        check(parse_image_header(junk)[1:] == (16, 16),
+              "junk fixture parses as a 16x16 WebP")
+        fd, webp_path = tempfile.mkstemp(suffix=".webp")
+        with os.fdopen(fd, "wb") as f:
+            f.write(junk)
+        d = ImageData(webp_path)
+        check(not d.ok and "conversion failed" in d.error,
+              "undecodable webp results in the error state")
+        os.unlink(webp_path)
+    finally:
+        if prev_cache_env is not None:
+            os.environ["NOMADNET_IMAGE_CACHE"] = prev_cache_env
+        else:
+            os.environ.pop("NOMADNET_IMAGE_CACHE", None)
+        shutil.rmtree(cache_tmp, ignore_errors=True)
+        _webp.reset()
+
+
+def test_winsize_parsing():
+    print("== TIOCGWINSZ parsing ==")
+    from nomadnet.ui.textui.images import _termlib
+    # struct order is (rows, cols, xpixel, ypixel)
+    check(_termlib._cell_size_from_winsize([40, 100, 1000, 800]) == (10, 20),
+          "correct winsize -> (10, 20)")
+    check(_termlib._cell_size_from_winsize([24, 80, 960, 480]) == (12, 20),
+          "correct winsize -> (12, 20)")
+    check(_termlib._cell_size_from_winsize([24, 80, 0, 0]) is None,
+          "zero pixel fields -> None")
+    check(_termlib._cell_size_from_winsize([40, 100, 800, 1000]) == (8, 25),
+          "portrait window cell keeps valid aspect")
+    check(_termlib._cell_size_from_winsize([30, 80, 1200, 240]) is None,
+          "implausible aspect (broken reporting) -> None")
+    check(_termlib._cell_size_from_winsize([0, 0, 0, 0]) is None,
+          "empty winsize -> None")
+
+
+def test_widget_layout(image_path):
+    print("== widget layout ==")
+    _termlib.kitty_supported = True
+    _termlib.cell_size = CELL_SIZE
+    # Force a deterministic terminal size for the height-percent cases.
+    _orig_terminal_size = _termlib.get_terminal_size
+    _termlib.get_terminal_size = lambda: (80, 24)
+    _orig_ui_margin = None
+    from nomadnet.ui.textui.images import widget as _widget_mod
+    _orig_ui_margin = _widget_mod.UI_MARGIN
+    _widget_mod.UI_MARGIN = 8
+    try:
+        imgdata = ImageData(image_path)
+        w = ImageWidget(image_path)
+        iw, ih = imgdata.size
+        expected_rows = max(1, round(80 * CELL_SIZE[0] * ih / (CELL_SIZE[1] * iw)))
+        rows = w.rows((80,))
+        check(rows == expected_rows,
+              "default: rows((80,)) == %d, got %d" % (expected_rows, rows))
+        canvas = w.render((80,))
+        check(isinstance(canvas, urwid.SolidCanvas), "render returns a solid canvas")
+        check(canvas.rows() == rows and canvas.cols() == 80, "canvas sized 80x%d" % rows)
+        check(w._ti_rows == rows and w._ti_cols == 80, "widget stores display size")
+        w.close()
+
+        # width as absolute columns (aspect preserved); the canvas fills the
+        # allocated width while the image rect is the resolved size.
+        w = ImageWidget(image_path, width=40)
+        c, r = w._display_size(80)
+        check((c, r) == (40, 15), "width=40 -> (40, 15), got (%d, %d)" % (c, r))
+        canvas = w.render((80,))
+        check((canvas.cols(), canvas.rows()) == (80, 15),
+              "width=40 canvas fills the row (80, 15), got (%d, %d)"
+              % (canvas.cols(), canvas.rows()))
+        check((w._ti_cols, w._ti_rows) == (40, 15),
+              "width=40 image rect stored (40, 15)")
+        w.close()
+
+        # width as percent of the layout width
+        w = ImageWidget(image_path, width="50%")
+        c, r = w._display_size(80)
+        check((c, r) == (40, 15), "width=50%% -> (40, 15), got (%d, %d)" % (c, r))
+        w.close()
+
+        # height as absolute rows (width derived, aspect preserved)
+        w = ImageWidget(image_path, height=20)
+        c, r = w._display_size(80)
+        check(r == 20 and c == 53, "height=20 -> (53, 20), got (%d, %d)" % (c, r))
+        w.close()
+
+        # height as percent of the visible page height
+        # page = terminal lines (24) - UI_MARGIN (8) = 16; 50% -> 8 rows
+        w = ImageWidget(image_path, height="50%")
+        c, r = w._display_size(80)
+        check(r == 8 and c == 21, "height=50%% -> (21, 8), got (%d, %d)" % (c, r))
+        w.close()
+
+        # both given: stretch to the exact rectangle
+        w = ImageWidget(image_path, width=40, height=30)
+        c, r = w._display_size(80)
+        check((c, r) == (40, 30), "width=40,height=30 stretches -> (40, 30), got (%d, %d)" % (c, r))
+        w.close()
+
+        # stretching is clamped uniformly when overflowing the space
+        w = ImageWidget(image_path, width=400, height=200)
+        c, r = w._display_size(80)
+        check((c, r) == (80, 40), "overflowing stretch clamped -> (80, 40), got (%d, %d)" % (c, r))
+        w.close()
+
+        # BOX layout: the canvas fills the box; the image rect is clamped
+        # and re-derived (width from the clamped height).
+        w = ImageWidget(image_path, name="box")
+        canvas = w.render((80, 10))
+        check((canvas.cols(), canvas.rows()) == (80, 10),
+              "BOX render((80,10)) canvas fills the box, got (%d, %d)"
+              % (canvas.cols(), canvas.rows()))
+        check((w._ti_cols, w._ti_rows) == (27, 10),
+              "BOX image rect clamped -> (27, 10), got (%d, %d)"
+              % (w._ti_cols, w._ti_rows))
+        w.close()
+
+        # invalid specifications
+        def raises(ctor, want):
+            try:
+                ctor()
+            except want:
+                return True
+            except Exception:
+                return False
+            return False
+        check(raises(lambda: ImageWidget(image_path, width="xx"), ValueError),
+              "width='xx' raises ValueError")
+        check(raises(lambda: ImageWidget(image_path, height=0), ValueError),
+              "height=0 raises ValueError")
+        check(raises(lambda: ImageWidget(image_path, width=1.5), TypeError),
+              "width=1.5 raises TypeError")
+        check(raises(lambda: ImageWidget(image_path, height="50"), ValueError),
+              "height='50' raises ValueError")
+
+        # settable properties re-resolve and invalidate
+        w = ImageWidget(image_path)
+        w.width = "50%"
+        c, r = w._display_size(80)
+        check((c, r) == (40, 15), "set width=50%% property -> (40, 15)")
+        w.height = 10
+        c, r = w._display_size(80)
+        check((c, r) == (40, 10), "width+height set -> stretch (40, 10), got (%d, %d)" % (c, r))
+        w.width = None
+        w.height = 10
+        c, r = w._display_size(80)
+        check((c, r) == (27, 10), "height-only after reset -> (27, 10), got (%d, %d)" % (c, r))
+        w.close()
+
+        # native resolution specification ("n"): display at the image's own
+        # pixel size, quantized to the terminal cell grid. Cell = (1, 2),
+        # small fixture 64x32 -> native 64 cols, 16 rows.
+        from nomadnet.ui.textui.images import _webp
+        small_png = _webp._encode_png_rgba(bytes(64 * 32 * 4), 64, 32)
+        fd, small_path = tempfile.mkstemp(suffix=".png")
+        with os.fdopen(fd, "wb") as f:
+            f.write(small_png)
+        try:
+            wn = ImageWidget(small_path, width="n")
+            wn.render((80,))  # sets _ti_cols/_ti_rows
+            check((wn._ti_cols, wn._ti_rows) == (64, 16),
+                  "width='n' displays native size (64, 16), got (%d, %d)"
+                  % (wn._ti_cols, wn._ti_rows))
+            wn.close()
+            hn = ImageWidget(small_path, height="n")
+            hn.render((80,))
+            check((hn._ti_cols, hn._ti_rows) == (64, 16),
+                  "height='n' displays native size (64, 16), got (%d, %d)"
+                  % (hn._ti_cols, hn._ti_rows))
+            hn.close()
+            bn = ImageWidget(small_path, width="n", height="n")
+            bn.render((80,))
+            check((bn._ti_cols, bn._ti_rows) == (64, 16),
+                  "width='n' height='n' displays native size (64, 16), got (%d, %d)"
+                  % (bn._ti_cols, bn._ti_rows))
+            bn.close()
+            big = ImageWidget(image_path, width="n")  # 1280x960, overflows
+            c, r = big._display_size(80)
+            check((c, r) == (80, 30),
+                  "native of a large image clamps to available width (80, 30), got (%d, %d)" % (c, r))
+            big.close()
+            check(raises(lambda: ImageWidget(image_path, width="x"), ValueError),
+                  "width='x' raises ValueError")
+        finally:
+            os.unlink(small_path)
+
+        # alignment: placement column of a narrowed image inside its canvas
+        w = ImageWidget(image_path, width=40)  # default center
+        w.render((80,))  # sets _ti_cols/_ti_rows
+        check(w._ti_align == "|", "default alignment is center")
+        key = screen_probe_placement_key(w, 3, 0, 80, 0, 15)
+        check(key == (3, 20, 40, 15, None, None),
+              "centered placement col -> %s" % (key,))
+        w.align = "left"
+        key = screen_probe_placement_key(w, 3, 0, 80, 0, 15)
+        check(key == (3, 0, 40, 15, None, None), "left placement col -> 0")
+        w.align = "right"
+        key = screen_probe_placement_key(w, 3, 0, 80, 0, 15)
+        check(key == (3, 40, 40, 15, None, None), "right placement col -> 40")
+        check(raises(lambda: ImageWidget(image_path, align="top"), ValueError),
+              "align='top' raises ValueError")
+        w.close()
+    finally:
+        _termlib.get_terminal_size = _orig_terminal_size
+        if _orig_ui_margin is not None:
+            _widget_mod.UI_MARGIN = _orig_ui_margin
+
+
+def find_image_widget(pile):
+    for widget, _opts in pile.contents:
+        child = widget
+        if isinstance(child, urwid.AttrMap):
+            child = child.original_widget
+        if isinstance(child, ImageWidget):
+            return child
+    return None
+
+
+def run_screen_test(image_path, wrap_attrmap, label):
+    print("== screen state machine (%s) ==" % label)
+    _termlib.kitty_supported = True
+    _termlib.cell_size = CELL_SIZE
+    _imagestore.image_store.reset()
+
+    pile = build_pile(image_path, wrap_attrmap=wrap_attrmap)
+    scrollable = urwid.Scrollable(pile, force_forward_keypress=True)
+    screen = RecordingScreen()
+
+    img_widget = find_image_widget(pile)
+    check(img_widget is not None, "image widget found in pile")
+    z = img_widget._ti_z
+    entry = _imagestore.image_store.entry_for(img_widget._ti_key)
+    check(entry is not None and entry["refs"] == 1, "widget registered with image store")
+    image_id = entry["image_id"]
+    b64 = base64.b64encode(img_widget._ti_data.data).decode("ascii")
+
+    def entries_at(offset):
+        scrollable.set_scrollpos(offset)
+        canvas = scrollable.render(VIEWPORT)
+        return screen._ti_scan(canvas)
+
+    # Build a timeline of image visibility across all scroll offsets.
+    # state = (trim_top, row, visible_rows)
+    timeline = []
+    max_offset = 200
+    for off in range(max_offset + 1):
+        entries = entries_at(off)
+        if not entries:
+            timeline.append((off, None))
+            continue
+        widget, row, col, canvas_cols, trim_top, vis_rows = entries[0]
+        timeline.append((off, (trim_top, row, vis_rows)))
+    image_rows = img_widget._ti_rows
+    visible = [t for t in timeline if t[1] is not None]
+    check(len(visible) > 0, "image is visible at some scroll offset")
+    if not visible:
+        return
+
+    def first_offset(pred):
+        for off, state in timeline:
+            if state is not None and pred(state):
+                return off
+        return None
+
+    off_enter = first_offset(lambda s: s[0] == 0)       # trim_top == 0
+    off_clip = first_offset(lambda s: s[0] > 0)         # top-clipped
+    check(off_enter is not None, "found offset with image fully entered")
+    check(off_clip is not None, "found offset with image top-clipped")
+    if off_enter is None or off_clip is None:
+        return
+    # State at the entry offset: (trim_top, row, visible_rows)
+    off_enter_vis_rows = timeline[off_enter][1][2]
+
+    last_visible_off = visible[-1][0]
+    off_away = last_visible_off + 1 if last_visible_off < max_offset else None
+    check(off_away is not None, "found offset past the image")
+
+    def update(offset):
+        screen.log = []
+        scrollable.set_scrollpos(offset)
+        canvas = scrollable.render(VIEWPORT)
+        screen._ti_update(canvas)
+        return list(screen.log)
+
+    # 1. before the image is visible: nothing
+    log = update(0)
+    check(not any("a=t" in l or "a=p" in l for l in log),
+          "nothing written while out of view")
+    check(not screen._ti_images, "nothing tracked while out of view")
+
+    # 2. image entered (bottom-clipped): transmit once, place the visible
+    #    slice (y=0 crop) at the bottom.
+    log = update(off_enter)
+    trans = extract_transmit(log)
+    placements = extract_placements(log)
+    check(trans is not None, "image data transmitted on entry")
+    if trans:
+        trans_id, pb64 = trans
+        check(trans_id == image_id, "transmit uses the store's image id")
+        check(pb64 == b64, "transmitted payload is the original file")
+    check(len(placements) == 1, "one placement on entry, got %d" % len(placements))
+    if placements:
+        ctrl = placements[0]
+        check("i=%d" % image_id in ctrl and "p=%d" % z in ctrl,
+              "placement references image id and placement id")
+        check("c=80" in ctrl and "r=%d" % off_enter_vis_rows in ctrl,
+              "placement has display size")
+        check("y=0," in ctrl and "h=" in ctrl, "entry slice uses a y=0 crop")
+        check("z=%d" % z in ctrl and "C=1" in ctrl and "q=1" in ctrl,
+              "placement has z-index, C=1 and quiet mode")
+    check(len(screen._ti_images) == 1 and
+          screen._ti_images[id(img_widget)]["drawn"] is not None,
+          "image tracked as drawn")
+    check(entry["transmitted"], "store entry marked transmitted")
+
+    # 3. move while visible: in-place placement update, no re-transmit,
+    #    no delete.
+    log = update(off_enter + 1)
+    check(extract_transmit(log) is None, "position change does not re-transmit")
+    check(len(extract_placements(log)) == 1, "position change re-places in place")
+    check(not extract_deletes(log), "position change does not delete")
+
+    # 4. top-clipped: placement with a y>0 crop.
+    log = update(off_clip)
+    placements = extract_placements(log)
+    check(len(placements) == 1, "top-clip places a slice, got %d" % len(placements))
+    if placements:
+        ctrl = placements[0]
+        y_px = int(ctrl.split("y=")[1].split(",")[0].split(";")[0])
+        check(",y=" in ctrl and "h=" in ctrl and y_px > 0,
+              "top-clipped placement has y>0 crop (y=%d)" % y_px)
+
+    # 5. scrolled past: placement deleted, image data kept.
+    log = update(off_away)
+    deletes = extract_deletes(log)
+    check(deletes == ["a=d,d=i,i=%d,p=%d" % (image_id, z)],
+          "scroll-away deletes the placement only: %s" % deletes)
+    check(extract_transmit(log) is None, "scroll-away does not transmit")
+    check(id(img_widget) not in screen._ti_images, "widget untracked after scroll-away")
+
+    # 6. scroll back: re-placed from the kept data, no re-transmission.
+    log = update(off_enter)
+    check(extract_transmit(log) is None, "scroll-back does not re-transmit")
+    check(len(extract_placements(log)) == 1, "scroll-back re-places the image")
+
+    # 7. still visible, unchanged position: nothing written.
+    log = update(off_enter)
+    check(not extract_placements(log) and not extract_deletes(log),
+          "unchanged position writes nothing")
+
+    # 8. widget closed -> orphaned data evicted from the terminal.
+    img_widget.close()
+    log = update(off_away)
+    deletes = extract_deletes(log)
+    check("a=d,d=I,i=%d" % image_id in deletes,
+          "orphaned image data evicted with d=I")
+    check(_imagestore.image_store.entry_for(img_widget._ti_key) is None,
+          "evicted entry dropped from the store")
+
+
+def test_eviction_and_purge(image_path):
+    print("== eviction and purge ==")
+    _termlib.kitty_supported = True
+    _termlib.cell_size = CELL_SIZE
+    _imagestore.image_store.reset()
+
+    # Shared source: two widgets with the same file share one store entry.
+    w1 = ImageWidget(image_path)
+    w2 = ImageWidget(image_path)
+    entry = _imagestore.image_store.entry_for(w1._ti_key)
+    check(entry["refs"] == 2, "identical sources share one registry entry (refs=%d)"
+          % entry["refs"])
+    check(w2._ti_key == w1._ti_key, "identical sources have the same content key")
+
+    w1.close()
+    check(entry["refs"] == 1, "closing one widget decrements refcount")
+    check(entry["transmitted"] is False, "no data transmitted yet")
+
+    # Place w2 via a screen update and verify orphan eviction after close.
+    pile = urwid.Pile([urwid.Text("x"), urwid.Divider(" "), w2,
+                       urwid.Divider(" "), urwid.Text("y")])
+    scrollable = urwid.Scrollable(pile, force_forward_keypress=True)
+    screen = RecordingScreen()
+    image_id = entry["image_id"]
+
+    def update(offset):
+        screen.log = []
+        scrollable.set_scrollpos(offset)
+        screen._ti_update(scrollable.render(VIEWPORT))
+        return list(screen.log)
+
+    log = update(2)
+    check(extract_transmit(log) is not None, "image transmitted when widget visible")
+    check(entry["transmitted"], "entry marked transmitted")
+
+    w2.close()
+    log = update(0)
+    deletes = extract_deletes(log)
+    check("a=d,d=I,i=%d" % image_id in deletes, "last reference closed -> d=I eviction")
+    check(_imagestore.image_store.entry_for(w2._ti_key) is None,
+          "entry dropped after eviction")
+
+    # Purge: after transmitting, purge_images() deletes everything and the
+    # next redraw re-transmits for still-visible widgets.
+    _imagestore.image_store.reset()
+    w3 = ImageWidget(image_path)
+    pile = urwid.Pile([urwid.Text("x"), urwid.Divider(" "), w3,
+                       urwid.Divider(" "), urwid.Text("y")])
+    scrollable = urwid.Scrollable(pile, force_forward_keypress=True)
+    screen2 = RecordingScreen()
+
+    def update2(offset):
+        screen2.log = []
+        scrollable.set_scrollpos(offset)
+        screen2._ti_update(scrollable.render(VIEWPORT))
+        return list(screen2.log)
+
+    log = update2(2)
+    check(extract_transmit(log) is not None, "purge-test: image transmitted")
+    screen2.log = []
+    screen2.purge_images()
+    check(any("a=d,d=A" in l for l in screen2.log), "purge deletes all (d=A)")
+    check(_imagestore.image_store.transmitted() == [], "purge marks all untransmitted")
+
+    log = update2(2)
+    check(extract_transmit(log) is not None, "visible image re-transmitted after purge")
+    check(len(extract_placements(log)) == 1, "visible image re-placed after purge")
+    w3.close()
+
+
+def main():
+    image_path = sys.argv[1] if len(sys.argv) > 1 else DEFAULT_IMAGE
+    print("Using image: %s" % image_path)
+    _termlib.reset_caches()
+    print("ambient cell size (env): %s" % (_termlib._query_cell_size(),))
+
+    test_header_parse(image_path)
+    test_transmission_encoding(image_path)
+    test_webp_conversion()
+    test_winsize_parsing()
+    test_widget_layout(image_path)
+    run_screen_test(image_path, wrap_attrmap=False, label="bare widget")
+    run_screen_test(image_path, wrap_attrmap=True, label="attrmap-wrapped")
+    test_eviction_and_purge(image_path)
+
+    print("\n%d passed, %d failed" % (PASS, FAIL))
+    return 1 if FAIL else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

nomadnet/ui/textui/images/_termlib.py

@@ -0,0 +1,199 @@
+# Terminal queries for capability detection and layout metrics.
+# 
+# All values are resolved once and cached; subsequent calls are instant.
+# Queries should be performed *before* starting the urwid main loop, because
+# they read directly from the terminal and would otherwise race with urwid's
+# own input handling. If no terminal is available, or the queries time out,
+# sensible fallbacks are used.
+
+import os
+import select
+import shutil
+import sys
+import time
+
+try: import termios
+except: termios = None
+
+from . import _ctlseqs
+
+QUERY_TIMEOUT = 0.3
+
+kitty_supported = None
+cell_size = None
+
+# Last raw terminal query response (bytes), for diagnostics.
+last_query_response = bytearray()
+last_kitty_response = bytearray()
+
+# Last TIOCGWINSZ buffer (rows, cols, xpixel, ypixel), for diagnostics.
+winsize = [0, 0, 0, 0]
+
+# Number of times a query attempt has been made; used to avoid repeated
+# slow queries in a broken environment.
+_query_failed = False
+
+def _open_tty():
+    # Opens the active terminal for reading and writing.
+    for stream in ("out", "in", "err"):
+        try: return os.open(os.ttyname(getattr(sys, "__std%s__" % stream).fileno()), os.O_RDWR)
+        except (OSError, AttributeError): continue
+    try: return os.open("/dev/tty", os.O_RDWR | os.O_NOCTTY)
+    except OSError: return -1
+
+
+def query_tty(request, timeout=QUERY_TIMEOUT):
+    # Sends a query to the terminal and returns the raw response.
+    global _query_failed
+    if _query_failed or not termios: return b""
+
+    fd = _open_tty()
+    if fd == -1:
+        _query_failed = True
+        return b""
+
+    # Never block indefinitely, even on a broken pseudo-terminal,
+    # select has a deadline and reads are non-blocking.
+    try: os.set_blocking(fd, False)
+    except OSError:  pass
+
+    old_attr = termios.tcgetattr(fd)
+    new_attr = termios.tcgetattr(fd)
+    # Raw-ish mode; disable canonical mode and echo. In canonical mode the
+    # terminal buffers input until a newline, which terminal query responses
+    # (for example. kitty's "i=31;OK") never contain. The read would block until
+    # timeout and the query would always "fail".
+    new_attr[3] &= ~(termios.ECHO | termios.ICANON)
+    new_attr[6][termios.VMIN] = 0
+    new_attr[6][termios.VTIME] = 0
+    try:
+        # TCSANOW (not TCSAFLUSH): TCSAFLUSH drains the output queue first,
+        # which can block indefinitely on some pseudo-terminals.
+        termios.tcsetattr(fd, termios.TCSANOW, new_attr)
+        try: os.write(fd, request)
+        except (BlockingIOError, OSError): return b""
+        response = bytearray()
+        deadline = time.monotonic() + timeout
+        while True:
+            remaining = deadline - time.monotonic()
+            if remaining <= 0: break
+            try: ready, _, _ = select.select([fd], [], [], remaining)
+            except (OSError, ValueError): break
+            if not ready: break
+            try: chunk = os.read(fd, 4096)
+            except (BlockingIOError, OSError): continue
+            if not chunk: break
+            response += chunk
+            if response.endswith(b"\x1b\\") or response.endswith(b"c"): break
+        last_query_response[:] = response
+        return bytes(response)
+    
+    finally:
+        try: termios.tcsetattr(fd, termios.TCSANOW, old_attr)
+        except termios.error: pass
+        finally: os.close(fd)
+
+def is_kitty_supported():
+    global kitty_supported
+    if kitty_supported is None:
+        response = query_tty(_ctlseqs.KITTY_QUERY)
+        last_kitty_response[:] = response
+        kitty_supported = bool(response and _ctlseqs.KITTY_QUERY_RESPONSE_RE.search(response))
+    return kitty_supported
+
+def force_kitty_supported(status):
+    # Forces the support status (used by tests).
+    global kitty_supported
+    kitty_supported = bool(status)
+
+def _cell_size_from_winsize(buf):
+    # Interprets a TIOCGWINSZ buffer as (cell_width_px, cell_height_px).
+    # The struct layout is (rows, cols, xpixel, ypixel). Returns None if the
+    # terminal does not report pixel dimensions, or if the result is
+    # implausible (e.g. several times wider than tall - a sign of broken
+    # winsize reporting).
+    rows, cols, xpix, ypix = buf
+    if not (cols and rows and xpix and ypix): return None
+    cell_w, cell_h = max(1, xpix // cols), max(1, ypix // rows)
+    if not (0.1 <= cell_w / cell_h <= 1.5):
+        # Real terminal cells should be as wide as, or narrower than,
+        # they are tall; anything far outside that range is bogus.
+        return None
+    return (cell_w, cell_h)
+
+def _query_cell_size():
+    # Determines the current cell size in pixels, or None if unknown.
+    if not termios: return None
+    fd = _open_tty()
+    if fd != -1:
+        # First, try the TIOCGWINSZ pixel fields (fast, no query)
+        import array
+        import fcntl
+        try:
+            buf = array.array("H", [0, 0, 0, 0])
+            fcntl.ioctl(fd, termios.TIOCGWINSZ, buf)
+            winsize[:] = buf
+            cell = _cell_size_from_winsize(buf)
+            if cell is not None: return cell
+        
+        except OSError: pass
+        finally: os.close(fd)
+
+    # Fall back to XTWINOPS 16 (cell size in pixels); DA1 is appended to
+    # speed up the response (terminals treat queries as FIFO).
+    response = query_tty(_ctlseqs.CELL_SIZE_PX + _ctlseqs.DA1)
+    if response:
+        match = _ctlseqs.CELL_SIZE_RESPONSE_RE.search(response)
+        if match: return (int(match.group("w")), int(match.group("h")))
+    return None
+
+def get_cell_size():
+    # Returns the terminal cell size in pixels as (width, height), or None
+    # if it could not be determined.
+    global cell_size
+    if cell_size is None: cell_size = _query_cell_size()
+    return cell_size
+
+
+def get_terminal_size():
+    # Returns the active terminal size as (columns, lines).
+    # Uses the tty directly when available (correct even with redirected
+    # output), falling back to shutil.get_terminal_size.
+    
+    fd = _open_tty()
+    if fd != -1:
+        try:
+            size = os.get_terminal_size(fd)
+            return (size.columns, size.lines)
+        except OSError: pass
+        finally: os.close(fd)
+    size = shutil.get_terminal_size(fallback=(80, 24))
+    return (size.columns, size.lines)
+
+def reset_caches():
+    # Clears all cached capability/metric values (used by tests).
+    global kitty_supported, cell_size, _query_failed
+    kitty_supported = None
+    cell_size = None
+    _query_failed = False
+    last_query_response[:] = b""
+    last_kitty_response[:] = b""
+    winsize[:] = (0, 0, 0, 0)
+
+def diagnostics():
+    # Returns a list of diagnostic strings about the terminal environment,
+    # for display/debugging.
+    import os
+    lines = [ "TERM=%r" % os.environ.get("TERM"),
+              "TERM_PROGRAM=%r TERM_PROGRAM_VERSION=%r" % (os.environ.get("TERM_PROGRAM"), os.environ.get("TERM_PROGRAM_VERSION")),
+              "kitty graphics protocol supported: %s" % is_kitty_supported(),
+              "kitty query response: %r" % bytes(last_kitty_response),
+              "cell size (px): %s" % (get_cell_size(),),
+              "cell size query response: %r" % bytes(last_query_response),
+              "winsize (rows, cols, xpix, ypix): %r" % (tuple(winsize),) ]
+    try:
+        from . import _webp
+        lines.append("webp conversion: %s" % _webp.describe())
+    except Exception: pass
+
+    return lines

nomadnet/ui/textui/images/_webp.py

@@ -0,0 +1,367 @@
+# Multi-backend WebP decoding
+# 
+# Kitty's graphics protocol decodes PNG natively but not WebP, so WebP
+# sources are converted transparently at the ImageData layer. This module
+# provides a conversion chain that *should* work almost anywhere with
+# zero extra dependencies:
+# 
+#  - Pillow/PIL, if installed with WebP support
+#  - libwebp via ctypes, decoding to RGBA, paired with a pure-Python PNG
+#    encoder. The system libwebp is present on most Linux/macOS systems
+#    as a transitive dependency of common graphics stacks and browsers
+#  - sips - the macOS built-in image tool
+#  - dwebp - from the system PATH (the "official" WebP tool)
+#  - ImageMagick / GraphicsMagick (magick, convert, gm convert)
+#  - ffmpeg / avconv
+#  - gdk-pixbuf-thumbnailer (GTK/GNOME systems)
+# 
+# Every backend's output is validated (it must be a valid PNG with the same
+# dimensions as the source WebP), so missing tools, or tools built without
+# WebP support, are simply skipped. The first backend that produces a valid
+# conversion is remembered and preferred for subsequent conversions, but the
+# preference is replaced whenever a different backend wins for a particular
+# image.
+# 
+# Environment overrides:
+# 
+#     NOMADNET_IMAGE_BACKEND   force a specific backend (strict)
+
+import ctypes
+import ctypes.util
+import hashlib
+import io
+import os
+import shutil
+import struct
+import subprocess
+import tempfile
+import zlib
+
+MAX_CACHE_BYTES = 192 * 1024 * 1024
+MAX_CACHE_FILES = 512
+CONVERSION_TIMEOUT = 5
+
+# libwebp soname candidates, in order of preference.
+_LIBWEBP_NAMES = [ "libwebp.so.7",
+                   "libwebp.so.6",
+                   "libwebp.so.5",
+                   "libwebp.so",
+                   "libwebp.dylib",
+                   "libwebp-7.dll",
+                   "libwebp.dll" ]
+
+# The built-in backend chain, in preference order.
+BACKENDS = ( "pil",
+             "libwebp-ctypes",
+             "sips",
+             "dwebp",
+             "magick",
+             "convert",
+             "gm",
+             "ffmpeg",
+             "avconv",
+             "gdk-pixbuf" )
+
+_ENV_BACKEND = os.environ.get("NOMADNET_IMAGE_BACKEND")
+_winner = None        # name of the currently preferred backend
+_last_backend = None  # backend used for the most recent conversion
+_convert_count = 0
+_cache_hits = 0
+
+_libwebp_cache = {"lib": None, "probed": False}
+
+###############################
+# PNG validation and encoding #
+###############################
+
+# Returns (width, height) if png is a valid PNG, else None
+def _png_dims(png):
+    if not png.startswith(b"\x89PNG\r\n\x1a\n") or len(png) < 24:
+        return None
+    if png[12:16] != b"IHDR":
+        return None
+    width = int.from_bytes(png[16:20], "big")
+    height = int.from_bytes(png[20:24], "big")
+    if not (width > 0 and height > 0):
+        return None
+    return (width, height)
+
+
+# Encodes RGBA pixel data as a PNG (8-bit RGBA, no filtering)
+def _encode_png_rgba(rgba, width, height):
+    def chunk(tag, payload):
+        return (struct.pack(">I", len(payload)) + tag + payload +
+                struct.pack(">I", zlib.crc32(tag + payload) & 0xFFFFFFFF))
+
+    ihdr = struct.pack(">IIBBBBB", width, height, 8, 6, 0, 0, 0)
+    stride = width * 4
+    raw = bytearray(height * (stride + 1))
+    pos = 0
+    for row in range(height):
+        raw[pos] = 0  # filter type: none
+        pos += 1
+        start = row * stride
+        raw[pos:pos + stride] = rgba[start:start + stride]
+        pos += stride
+    
+    return (b"\x89PNG\r\n\x1a\n" +
+            chunk(b"IHDR", ihdr) +
+            chunk(b"IDAT", zlib.compress(bytes(raw), 9)) +
+            chunk(b"IEND", b""))
+
+
+####################
+# Conversion cache #
+####################
+
+CONV_CACHE_BASE = None
+def cache_dir():
+    global CONV_CACHE_BASE
+    if not CONV_CACHE_BASE:
+        from nomadnet.NomadNetworkApp import NomadNetworkApp
+        CONV_CACHE_BASE = NomadNetworkApp._shared_instance.dispcachepath
+    return CONV_CACHE_BASE
+
+def _cache_path(source_data):
+    return os.path.join(cache_dir(), hashlib.sha256(source_data).hexdigest() + ".png")
+
+def _cache_get(source_data):
+    try:
+        with open(_cache_path(source_data), "rb") as f: png = f.read()
+        if _png_dims(png) is not None: return png
+    except OSError: pass
+    return None
+
+def _cache_put(source_data, png):
+    try:
+        directory = cache_dir()
+        os.makedirs(directory, exist_ok=True)
+        fd, tmp_path = tempfile.mkstemp(prefix=".tmp-", dir=directory)
+        try:
+            with os.fdopen(fd, "wb") as f: f.write(png)
+            os.replace(tmp_path, _cache_path(source_data))
+        except OSError:
+            try: os.unlink(tmp_path)
+            except OSError: pass
+        _prune_cache()
+    except OSError: pass
+
+# Evicts least-recently-used entries beyond the size/file limits
+def _prune_cache():
+    try:
+        entries = []
+        for name in os.listdir(cache_dir()):
+            path = os.path.join(cache_dir(), name)
+            try:
+                st = os.stat(path)
+                if st.st_size > 0:
+                    entries.append((st.st_mtime, st.st_size, path))
+            except OSError: pass
+
+        total = sum(e[1] for e in entries)
+        entries.sort()
+        while len(entries) > MAX_CACHE_FILES or total > MAX_CACHE_BYTES:
+            _, size, path = entries.pop(0)
+            total -= size
+            try: os.unlink(path)
+            except OSError: pass
+    except OSError: pass
+
+#######################
+# Individual backends #
+#######################
+
+# Runs a conversion CLI over temp files
+def _run_cli(argv, data):
+    if shutil.which(argv[0]) is None: return None
+    try:
+        with tempfile.TemporaryDirectory(prefix="nomadnet-img-") as td:
+            src = os.path.join(td, "source.webp")
+            dst = os.path.join(td, "output.png")
+            with open(src, "wb") as f: f.write(data)
+            args = [a.replace("{src}", src).replace("{dst}", dst) for a in argv]
+            subprocess.run(
+                args,
+                timeout=CONVERSION_TIMEOUT,
+                # stdin must be closed: some tools (e.g. ffmpeg) block
+                # on inherited stdin otherwise.
+                stdin=subprocess.DEVNULL,
+                stdout=subprocess.DEVNULL,
+                stderr=subprocess.DEVNULL,
+            )
+            with open(dst, "rb") as f: return f.read()
+    except (OSError, subprocess.TimeoutExpired, ValueError): return None
+
+def _backend_pil(data, dims):
+    try:
+        from PIL import Image
+        from PIL import features
+        if not features.check("webp"): return None
+    except ImportError: return None
+    try:
+        image = Image.open(io.BytesIO(data))
+        image.load()
+        buf = io.BytesIO()
+        image.save(buf, format="PNG")
+        return buf.getvalue()
+    except Exception: return None
+
+# Loads the system libwebp via ctypes, caching the result.
+def _get_libwebp():
+    if _libwebp_cache["probed"]: return _libwebp_cache["lib"]
+    _libwebp_cache["probed"] = True
+    lib = None
+    names = list(_LIBWEBP_NAMES)
+    found = ctypes.util.find_library("webp")
+    if found and found not in names: names.insert(0, found)
+    for name in names:
+        try:
+            lib = ctypes.CDLL(name)
+            break
+        except OSError: continue
+    
+    if lib is not None:
+        lib.WebPGetInfo.restype = ctypes.c_int
+        lib.WebPGetInfo.argtypes = [ ctypes.c_char_p, ctypes.c_size_t,
+                                     ctypes.POINTER(ctypes.c_int), ctypes.POINTER(ctypes.c_int) ]
+
+        lib.WebPDecodeRGBA.restype = ctypes.c_void_p
+        lib.WebPDecodeRGBA.argtypes = [ ctypes.c_char_p, ctypes.c_size_t,
+                                        ctypes.POINTER(ctypes.c_int), ctypes.POINTER(ctypes.c_int) ]
+        lib.WebPFree.restype = None
+        lib.WebPFree.argtypes = [ctypes.c_void_p]
+
+    _libwebp_cache["lib"] = lib
+    return lib
+
+def _backend_ctypes(data, dims):
+    lib = _get_libwebp()
+    if lib is None: return None
+    try:
+        buf = ctypes.create_string_buffer(data, len(data))
+        width = ctypes.c_int(0)
+        height = ctypes.c_int(0)
+        if not lib.WebPGetInfo(buf, len(data), ctypes.byref(width), ctypes.byref(height)): return None
+        if width.value <= 0 or height.value <= 0: return None
+        ptr = lib.WebPDecodeRGBA(buf, len(data), ctypes.byref(width), ctypes.byref(height))
+        if not ptr: return None
+        try: rgba = ctypes.string_at(ptr, width.value * height.value * 4)
+        finally: lib.WebPFree(ptr)
+        return _encode_png_rgba(rgba, width.value, height.value)
+    except Exception: return None
+
+def _backend_gdkpixbuf(data, dims):
+    if dims is None: return None
+    width, height = dims
+    return _run_cli(["gdk-pixbuf-thumbnailer", "-s", "%dx%d" % (width, height), "{src}", "{dst}"], data)
+
+def _backend_sips(data, dims):    return _run_cli(["sips", "-s", "format", "png", "{src}", "--out", "{dst}"], data)
+def _backend_dwebp(data, dims):   return _run_cli(["dwebp", "{src}", "-o", "{dst}"], data)
+def _backend_magick(data, dims):  return _run_cli(["magick", "{src}", "{dst}"], data)
+def _backend_convert(data, dims): return _run_cli(["convert", "{src}", "{dst}"], data)
+def _backend_gm(data, dims):      return _run_cli(["gm", "convert", "{src}", "{dst}"], data)
+def _backend_ffmpeg(data, dims):  return _run_cli(["ffmpeg", "-y", "-loglevel", "error", "-i", "{src}", "-frames:v", "1", "{dst}"], data)
+def _backend_avconv(data, dims):  return _run_cli(["avconv", "-y", "-loglevel", "error", "-i", "{src}", "-frames:v", "1", "{dst}"], data)
+
+_BACKENDS = { "pil": _backend_pil,
+              "libwebp-ctypes": _backend_ctypes,
+              "sips": _backend_sips,
+              "dwebp": _backend_dwebp,
+              "magick": _backend_magick,
+              "convert": _backend_convert,
+              "gm": _backend_gm,
+              "ffmpeg": _backend_ffmpeg,
+              "avconv": _backend_avconv,
+              "gdk-pixbuf": _backend_gdkpixbuf }
+
+#####################
+# Conversion driver #
+#####################
+
+def _webp_dims(data):
+    from ._imagedata import parse_image_header
+    try: fmt, width, height = parse_image_header(data)
+    except ValueError: return None
+    if fmt != "WebP": return None
+    return (width, height)
+
+def _backend_order():
+    if _ENV_BACKEND: return [_ENV_BACKEND] if _ENV_BACKEND in BACKENDS else []
+    order = list(BACKENDS)
+    if _winner in order:
+        order.remove(_winner)
+        order.insert(0, _winner)
+    return order
+
+# Converts WebP data to PNG bytes, or None on failure
+def convert_webp_to_png(data):
+    global _cache_hits
+    png = _cache_get(data)
+    if png is not None:
+        _cache_hits += 1
+        return png
+    return _convert_uncached(data)
+
+def _convert_uncached(data):
+    global _convert_count, _winner, _last_backend
+    dims = _webp_dims(data)
+    if dims is None: return None
+    for name in _backend_order():
+        try: png = _BACKENDS[name](data, dims)
+        except Exception: png = None
+        if png is not None and _png_dims(png) == dims:
+            _convert_count += 1
+            _last_backend = name
+            if not _ENV_BACKEND: _winner = name
+            _cache_put(data, png)
+            return png
+
+    return None
+
+##################################
+# Introspection and test support #
+##################################
+
+# Returns [(name, available), ...] for the built-in chain
+def available_backends():
+    out = []
+    for name in BACKENDS:
+        if name == "pil":
+            try:
+                from PIL import features
+                out.append((name, bool(features.check("webp"))))
+            except Exception: out.append((name, False))
+        elif name == "libwebp-ctypes": out.append((name, _get_libwebp() is not None))
+        elif name == "gdk-pixbuf":     out.append((name, shutil.which("gdk-pixbuf-thumbnailer") is not None))
+        else:                          out.append((name, shutil.which(name) is not None))
+
+    return out
+
+# A diagnostic string describing the conversion setup
+def describe():
+    if _ENV_BACKEND: return "backend forced via NOMADNET_IMAGE_BACKEND: %s" % _ENV_BACKEND
+    present = [name for name, avail in available_backends() if avail]
+    if not present: return "no conversion backend available (WebP images will not render)"
+    return "backend(s) available: %s; preferred: %s" % (", ".join(present), _last_backend or _winner or "none yet")
+
+# Usage statistics for diagnostics
+def stats():
+    return { "backend": _last_backend, "conversions": _convert_count,
+             "cache_hits": _cache_hits, "cache_dir": cache_dir() }
+
+def last_backend(): return _last_backend
+
+# Forces a specific backend name (testing/debugging); None to clear.
+def force_backend(name):
+    global _ENV_BACKEND
+    _ENV_BACKEND = name if name in BACKENDS else None
+
+# Clears dynamic state (winner, counters); used by tests.
+def reset():
+    global _winner, _last_backend, _convert_count, _cache_hits
+    _winner = None
+    _last_backend = None
+    _convert_count = 0
+    _cache_hits = 0
+    _libwebp_cache["lib"] = None
+    _libwebp_cache["probed"] = False

nomadnet/ui/textui/images/screen.py

@@ -0,0 +1,210 @@
+import urwid
+
+from . import _ctlseqs
+from ._imagestore import image_store
+from ._termlib import is_kitty_supported
+from .widget import ImageWidget
+
+# A raw display screen subclass with Kitty TGP side-channel support
+class ImageScreen(urwid.raw_display.Screen):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._ti_images = {}
+
+    ##########
+    # Output #
+    ##########
+
+    def write(self, data):
+        if isinstance(data, bytes): data = data.decode("ascii")
+        return super().write(data)
+
+    #############
+    # Lifecycle #
+    #############
+
+    def clear(self):
+        # Clears all images and forces a full repaint
+        if is_kitty_supported():
+            self.write(_ctlseqs.KITTY_DELETE_ALL)
+            self._ti_images = {}
+            image_store.mark_all_untransmitted()
+        return super().clear()
+
+    def _start(self, *args, **kwargs):
+        ret = super()._start(*args, **kwargs)
+        if is_kitty_supported():
+            # Clear any images left over from a previous run/instance.
+            self.write(_ctlseqs.KITTY_DELETE_ALL)
+            self._ti_images = {}
+            image_store.mark_all_untransmitted()
+        return ret
+
+    def _stop(self):
+        self.purge_images()
+        try: return super()._stop()
+        finally: self._ti_images = {}
+
+    def purge_images(self):
+        # Purges all image data from the terminal buffer.
+        if not is_kitty_supported(): return
+        if self._ti_images or image_store.transmitted():
+            self.write(_ctlseqs.KITTY_DELETE_ALL)
+        self._ti_images = {}
+        image_store.mark_all_untransmitted()
+
+    ###########
+    # Drawing #
+    ###########
+
+    def draw_screen(self, size, canvas):
+        # Draws the canvas, then updates on-screen images within a
+        # synchronized-update region to prevent flicker.        
+        self.write(_ctlseqs.BEGIN_SYNCED_UPDATE)
+        try:
+            super().draw_screen(size, canvas)
+            self._ti_update(canvas)
+        finally:
+            self.write(_ctlseqs.END_SYNCED_UPDATE)
+            self.flush()
+
+    def _ti_update(self, canvas):
+        # Places/updates/deletes image placements to match the current
+        # canvas, and evicts orphaned image data.
+        if not is_kitty_supported(): return
+
+        entries = self._ti_scan(canvas)
+        current = {}
+        for widget, row, col, canvas_cols, trim_top, vis_rows in entries:
+            current[id(widget)] = (widget, row, col, canvas_cols, trim_top, vis_rows)
+
+        # For widgets that scrolled completely out of view, remove their
+        # placement only. The image data is kept in the terminal so that
+        # scrolling back just re-places it.
+        for wid, info in list(self._ti_images.items()):
+            if wid in current: continue
+            self._ti_unplace(info["widget"])
+            del self._ti_images[wid]
+
+        # For visible widgets, place/update the visible slice in place.
+        for wid, (widget, row, col, canvas_cols, trim_top, vis_rows) in current.items():
+            info = self._ti_images.setdefault(wid, {"widget": widget, "drawn": None})
+            if vis_rows <= 0 or trim_top >= widget._ti_rows:
+                if info["drawn"] is not None:
+                    self._ti_unplace(widget)
+                    info["drawn"] = None
+                continue
+
+            key = self._placement_key(widget, row, col, canvas_cols, trim_top, vis_rows)
+            if key is None:
+                if info["drawn"] is not None:
+                    self._ti_unplace(widget)
+                    info["drawn"] = None
+                continue
+
+            if info["drawn"] == key: continue
+            self._ti_place(widget, key)
+            info["drawn"] = key
+
+        # Evict image data that no live widget references any more
+        for key, image_id in image_store.evict_orphans():
+            self.write(_ctlseqs.KITTY_DELETE_IMAGE % image_id)
+            image_store.evict(key)
+
+    #############
+    # Internals #
+    #############
+
+    def _ti_scan(self, canvas):
+        # Walks the canvas shards and returns a list of
+        # (widget, row, col, canvas_cols, trim_top, visible_rows)
+        # for every visible image canvas.
+        found = []
+        if not isinstance(canvas, urwid.CompositeCanvas): return found
+
+        def process_shard_tails():
+            nonlocal col
+            while col in shard_tails:
+                *trim, cols, rows, canv = shard_tails[col]
+                if rows > n_rows: shard_tails[col] = (*trim, cols, rows - n_rows, canv)
+                else:             del shard_tails[col]
+                col += cols
+
+        shard_tails = {}
+        row = 0
+        for n_rows, cviews in canvas.shards:
+            col = 0
+            for cview in cviews:
+                process_shard_tails()
+                *trim, cols, rows, _, canv = cview
+                widget = self._ti_widget_of(canv)
+                if widget is not None: found.append((widget, row, col, cols, trim[1], rows))
+                if rows > n_rows: shard_tails[col] = (*trim, cols, rows - n_rows, canv)
+                col += cols
+
+            process_shard_tails()
+            row += n_rows
+
+        return found
+
+    @staticmethod
+    def _ti_widget_of(canv):
+        # Returns the ImageWidget for a leaf canvas, or None
+        widget_info = getattr(canv, "_widget_info", None)
+        if not widget_info: return None
+        widget = widget_info[0]
+        if isinstance(widget, ImageWidget) and widget._ti_data.ok and not widget._ti_closed: return widget
+        return None
+
+    def _placement_key(self, widget, row, col, canvas_cols, trim_top, vis_rows):
+        # Compute placement parameters for the visible slice of
+        # an image, including the alignment offset.
+
+        cols = widget._ti_cols
+        rows = widget._ti_rows
+        if not (cols and rows): return None
+        place_col = col
+        if cols < canvas_cols:
+            align = widget._ti_align
+            spare = canvas_cols - cols
+            if align == ">":   place_col += spare
+            elif align == "|": place_col += spare // 2
+
+        vis_rows = max(1, min(vis_rows, rows))
+        y_px = h_px = None
+        if not (trim_top == 0 and vis_rows == rows):
+            # Crop to the visible slice of the image, in image pixels
+            ih = widget._ti_data.height
+            y_px = min(ih - 1, round(trim_top * ih / rows))
+            h_px = max(1, min(ih - y_px, round(vis_rows * ih / rows)))
+
+        return (row, place_col, cols, vis_rows, y_px, h_px)
+
+    def _ti_place(self, widget, key):
+        # Place (or in-place update) the visible slice of an image
+        row, place_col, cols, vis_rows, y_px, h_px = key
+        entry = image_store.entry_for(widget._ti_key)
+        if entry is None:
+            # This should not happen, but if the image data was evicted
+            # while the image was still alive, re-register it so the
+            # data can be re-transmitted to the terminal buffer.
+            image_store.register(widget._ti_key, widget._ti_data.data)
+            entry = image_store.entry_for(widget._ti_key)
+        image_id = entry["image_id"]
+
+        if not entry["transmitted"]:
+            payload = widget._ti_data.data
+            b64_len = (len(payload) * 4 + 2) // 3
+            n_chunks = max(1, (b64_len + _ctlseqs.CHUNK_SIZE - 1) // _ctlseqs.CHUNK_SIZE)
+            self.write(_ctlseqs.kitty_transmission(_ctlseqs.kitty_transmit_control(image_id), payload))
+            entry["transmitted"] = True
+
+        self.write(_ctlseqs.CURSOR_POSITION % (row + 1, place_col + 1))
+        self.write(_ctlseqs.kitty_placement(image_id, widget._ti_z, cols, vis_rows, widget._ti_z, y=y_px, h=h_px))
+
+    def _ti_unplace(self, widget):
+        # Delete a widget's placement, but keep the
+        # image data in the terminal buffer.
+        entry = image_store.entry_for(widget._ti_key)
+        if entry is None: return
+        self.write(_ctlseqs.KITTY_DELETE_PLACEMENT % (entry["image_id"], widget._ti_z))

nomadnet/ui/textui/images/widget.py

@@ -0,0 +1,283 @@
+# An urwid widget that occupies the space an image should fill in a layout,
+# without embedding any escape sequences in its canvas. The actual terminal
+# graphics are transmitted by the ImageScreen class which tracks the widget's
+# on-screen position after every draw.
+# 
+# This separation keeps the canvas trivially simple (urwid sees only blank
+# rows) and lets the screen handle the hard parts; positioning, deletion when
+# scrolled away, terminal resize and redraws - all of which are driven by
+# urwid's own shard bookkeeping rather than by fragile escape-sequence
+# manipulation inside canvas rows.
+# 
+# When the Kitty graphics protocol is unavailable (or the image cannot be
+# loaded), the widget renders a simple text placeholder instead.
+
+import os
+import re
+import urwid
+
+from . import _termlib
+from ._imagestore import image_store
+
+DEFAULT_CELL_SIZE = (1, 2)  # Fallback, assume a 1:2 cell aspect ratio
+MAX_Z_INDEX       = 2 ** 31 - 1
+UI_MARGIN         = 8
+_SIZE_SPEC_RE     = re.compile(r"(\d+(?:\.\d+)?)%")
+_ALIGNS           = { "left": "<", "<": "<",
+                      "center": "|", "|": "|",
+                      "right": ">", ">": ">" }
+
+def _check_align(value):
+    if value not in _ALIGNS: raise ValueError("invalid alignment %r (expected 'left', 'center' or 'right')" % (value,))
+    return _ALIGNS[value]
+
+
+def _check_size_spec(value, name):
+    # None = auto, int = absolute cells/rows, "NN%" = percent, "n" = native resolution
+    if value is None: return None
+    
+    if isinstance(value, int):
+        if value <= 0: raise ValueError("%s must be a positive integer, got %r" % (name, value))
+        return value
+    
+    if isinstance(value, str):
+        if value.lower() == "n": return "n"
+        match = _SIZE_SPEC_RE.fullmatch(value)
+        if not match or float(match.group(1)) <= 0:
+            raise ValueError("invalid %s specification %r (expected a positive integer, a percent string like '50%%' or 'n' for native resolution)" % (name, value))
+        return value
+    
+    raise TypeError("invalid type for %s (got %s)" % (name, type(value).__name__))
+
+
+class ImageWidget(urwid.Widget):
+    # A widget that displays an image via the Kitty graphics protocol.
+    # The widget is a plain FLOW/BOX widget; its canvas is blank. The
+    # ImageScreen class performs the actual drawing over the Kitty
+    # TGP side-channel.
+
+    placeholder_glyph = ""
+
+    _sizing = frozenset((urwid.FLOW, urwid.BOX))
+    ignore_focus = True
+
+    # Z-index allocation. Each widget instance gets a unique z-index so that
+    # the screen can delete exactly one image when it scrolls away or moves.
+    _ti_next_z = 1
+    _ti_free_z = set()
+
+    def __init__(self, path, name=None, width=None, height=None, align="center"):
+        if path and not isinstance(path, (str, os.PathLike)): raise TypeError("path must be a string or path-like object")
+        self.path = path
+        self._notice = None
+        self._ti_path = None
+        self._ti_data = None
+        self._ti_width = _check_size_spec(width, "width")
+        self._ti_height = _check_size_spec(height, "height")
+        self._ti_align = _check_align(align)
+        self._ti_name = name if name is not None else ""
+        self._ti_z = self._ti_alloc_z()
+        self.image_id = self._ti_z
+        
+        # Display size (cols, rows) from the most recent render.
+        # Used by the screen to place the image.
+        self._ti_cols = None
+        self._ti_rows = None
+        self._ti_closed = False
+        self._ti_key = None
+        super().__init__()
+        if self.path: self.load()
+
+    def load(self, path=None, remote_source=False):
+        if path: self.path = path
+        if self.path:
+            from ._imagedata import ImageData
+            self._ti_path = os.path.abspath(os.fspath(self.path))
+            self._ti_data = ImageData(self._ti_path, remote_source=remote_source)
+
+            if self._ti_data.ok:
+                # Register with the image store (de-duplicates identical data
+                # and allocates the terminal-side image id on first use).
+                self._ti_key = self._ti_data.key
+                image_store.register(self._ti_key, self._ti_data.data)
+                self._invalidate()
+            else:
+                error_msg = self._ti_data.error or "Unknown error"
+                self.notice(f"Error: {error_msg}")
+
+    def notice(self, msg=""):
+        if not msg: self._notice = None
+        else:       self._notice = str(msg)
+        self._invalidate()
+
+    def __del__(self):
+        try: type(self)._ti_free_z.add(self._ti_z)
+        except Exception: pass
+        try: self.close()
+        except Exception: pass
+
+    def close(self):
+        # Releases the widget's reference to its image data.
+        # The data is evicted from the terminal by the screen as soon as no
+        # live widget references it any more. Safe to call multiple times and
+        # from __del__.
+        try:
+            if not self._ti_closed:
+                self._ti_closed = True
+                if self._ti_key is not None: image_store.unregister(self._ti_key)
+        except Exception: pass
+
+    @classmethod
+    def _ti_alloc_z(cls):
+        if cls._ti_free_z: return cls._ti_free_z.pop()
+        z = cls._ti_next_z
+        if z >= MAX_Z_INDEX: raise RuntimeError("Too many image widgets")
+        cls._ti_next_z += 1
+        return z
+
+    ##########
+    # Sizing #
+    ##########
+
+    width = property(
+        lambda self: self._ti_width,
+        doc="""Display width specification: None (full width), a positive
+        int (columns), a percent string ("NN%") of the available layout
+        width, or "n" for native resolution.""",
+    )
+
+    @width.setter
+    def width(self, value):
+        self._ti_width = _check_size_spec(value, "width")
+        self._invalidate()
+
+    height = property(
+        lambda self: self._ti_height,
+        doc="""Display height specification: None (derived, aspect
+        preserving), a positive int (rows), a percent string ("NN%") of
+        the visible page height, or "n" for native resolution.""",
+    )
+
+    @height.setter
+    def height(self, value):
+        self._ti_height = _check_size_spec(value, "height")
+        self._invalidate()
+
+    align = property(
+        lambda self: self._ti_align,
+        doc="""Horizontal alignment of the image within its allocated
+        space: ``"<"`` (left), ``"|"`` (center) or ``">"`` (right).""",
+    )
+
+    @align.setter
+    def align(self, value):
+        self._ti_align = _check_align(value)
+        self._invalidate()
+
+    def _short(self): return "img#%x" % (id(self) & 0xFFFFF)
+
+    def _display_size(self, maxcol, maxrow=None):
+        if not self._ti_data.ok: return (min(maxcol, 1), 1)
+        iw, ih = self._ti_data.size
+        cell_w, cell_h = _termlib.get_cell_size() or DEFAULT_CELL_SIZE
+        w_spec, h_spec = self._ti_width, self._ti_height
+
+        def _pct(value): return float(value[:-1]) / 100.0
+
+        def _raw_width():
+            if w_spec is None: return float(maxcol)
+            if isinstance(w_spec, int): return float(w_spec)
+            if w_spec == "n": return iw / cell_w
+            return maxcol * _pct(w_spec)
+
+        def _raw_height():
+            if h_spec is None: return None
+            if isinstance(h_spec, int): return float(h_spec)
+            if h_spec == "n": return ih / cell_h
+            columns, lines = _termlib.get_terminal_size()
+            page = max(1, lines - UI_MARGIN)
+            return page * _pct(h_spec)
+
+        def _height_for_width(cols): return cols * cell_w * ih / (cell_h * iw)
+        def _width_for_height(rows): return rows * cell_h * iw / (cell_w * ih)
+
+        if w_spec is not None and h_spec is not None:
+            # Stretch mode: Target rectangle, uniformly scaled down only if
+            # it would overflow the available space.
+            target_w = max(1.0, _raw_width())
+            target_h = max(1.0, _raw_height())
+            scale = min(1.0, maxcol / target_w)
+            if maxrow is not None: scale = min(scale, maxrow / target_h)
+            cols = max(1, round(target_w * scale))
+            rows = max(1, round(target_h * scale))
+        elif w_spec is not None:
+            # Width given: Height derived, aspect preserving.
+            cols = max(1, min(maxcol, round(_raw_width())))
+            rows = max(1, round(_height_for_width(cols)))
+            if maxrow is not None and rows > maxrow:
+                rows = maxrow
+                cols = max(1, min(maxcol, round(_width_for_height(rows))))
+        elif h_spec is not None:
+            # Height given: Width derived, aspect preserving.
+            rows = max(1, round(_raw_height()))
+            cols = max(1, min(maxcol, round(_width_for_height(rows))))
+            if maxrow is not None and rows > maxrow:
+                rows = maxrow
+                cols = max(1, min(maxcol, round(_width_for_height(rows))))
+        else:
+            # Default: Full width, height derived.
+            cols = maxcol
+            rows = max(1, round(_height_for_width(cols)))
+            if maxrow is not None and rows > maxrow:
+                rows = maxrow
+                cols = max(1, min(maxcol, round(_width_for_height(rows))))
+        
+        return (cols, rows)
+
+    def rows(self, size, focus=False):
+        maxcol = size[0]
+        if not (self._ti_data and self._ti_data.ok and _termlib.is_kitty_supported()):
+            return len(self._placeholder_lines())
+        n = self._display_size(maxcol)[1]
+        return n
+
+    def render(self, size, focus=False):
+        if len(size) == 2:
+            maxcol, maxrow = size
+            is_box = True
+        else:
+            maxcol, maxrow = size[0], None
+            is_box = False
+
+        if not (self._ti_data and self._ti_data.ok and _termlib.is_kitty_supported()):
+            return self._placeholder_canvas(maxcol, maxrow, focus)
+
+        cols, rows = self._display_size(maxcol, maxrow)
+        self._ti_cols, self._ti_rows = cols, rows
+        # urwid requires canvases to fill the allocated space exactly
+        # (width == maxcol for FLOW, (maxcol, maxrow) for BOX). The image
+        # itself is placed by the screen at its resolved size inside the
+        # canvas, left-aligned; the rest of the canvas stays blank.
+        if is_box: return ImageCanvas(maxcol, maxrow)
+        else:      return ImageCanvas(maxcol, rows)
+
+    #########################
+    # Placeholder rendering #
+    #########################
+
+    def _placeholder_lines(self):
+        return [f"{self.placeholder_glyph}{self._notice or self._ti_name}"]
+
+    def _placeholder_canvas(self, maxcol, maxrow, focus):
+        align = {"<": "left", "|": "center", ">": "right"}.get(self._ti_align, "center")
+        text = urwid.Text("\n".join(self._placeholder_lines()), align=align)
+        if maxrow is not None: return urwid.Filler(text).render((maxcol, maxrow), focus)
+        else:                  return text.render((maxcol,), focus)
+
+
+class ImageCanvas(urwid.SolidCanvas):
+    # A blank canvas marking the space occupied by an image.
+    # Urwid sets _widget_info on this canvas to the owning
+    # ImageWidget. ImageScreen finds image canvases through
+    # that reference during its shard scan.
+    def __init__(self, cols, rows): super().__init__(" ", cols, rows)

nomadnet/util.py

@@ -0,0 +1,222 @@
+import os
+import re
+import unicodedata
+
+
+def write_config_atomic(config):
+    path = getattr(config, "filename", None)
+    if not path:
+        config.write()
+        return
+
+    directory = os.path.dirname(path) or "."
+    tmp = os.path.join(directory, "."+os.path.basename(path)+".tmp")
+    try:
+        try:
+            existing_mode = os.stat(path).st_mode
+        except OSError:
+            existing_mode = None
+
+        with open(tmp, "wb") as tmp_file:
+            config.write(tmp_file)
+            tmp_file.flush()
+            try: os.fsync(tmp_file.fileno())
+            except Exception: pass
+
+        if existing_mode is not None:
+            try: os.chmod(tmp, existing_mode)
+            except Exception: pass
+
+        os.replace(tmp, path)
+    except Exception:
+        try: os.remove(tmp)
+        except Exception: pass
+        raise
+
+
+invalid_rendering = ["🕵️", "☝"]
+
+# Unicode blocks to strip (symbols, dingbats, emoji ranges)
+STRIP_BLOCKS_RE = re.compile(
+    '['
+    '\U0001F600-\U0001F64F'  # Emoticons
+    '\U0001F300-\U0001F5FF'  # Misc Symbols & Pictographs
+    '\U0001F680-\U0001F6FF'  # Transport & Map Symbols
+    '\U0001F700-\U0001F77F'  # Alchemical Symbols
+    '\U0001F780-\U0001F7FF'  # Geometric Shapes Extended
+    '\U0001F800-\U0001F8FF'  # Supplemental Arrows-C
+    '\U0001F900-\U0001F9FF'  # Supplemental Symbols & Pictographs
+    '\U0001FA00-\U0001FA6F'  # Chess Symbols
+    '\U0001FA70-\U0001FAFF'  # Symbols & Pictographs Extended-A
+    '\U0001F1E0-\U0001F1FF'  # Flags (iOS/regional indicators)
+    '\u2600-\u26FF'          # Misc Symbols (☀, ☁, ☂, etc.)
+    '\u2700-\u27BF'          # Dingbats (✂, ✈, ✉, ✌, etc.)
+    '\uFE00-\uFE0F'          # Variation Selectors
+    '\U000E0100-\U000E01EF'  # Variation Selectors Supplement
+    '\U0001F3FB-\U0001F3FF'  # Emoji modifiers (skin tones)
+    ']+',
+    flags=re.UNICODE
+)
+
+# Control characters and zero-width characters to strip
+STRIP_CONTROL_RE = re.compile(
+    '['
+    '\x00-\x08'      # C0 controls (NUL-BS)
+    '\x0B\x0C'       # VT, FF
+    '\x0E-\x1F'      # C0 controls (SO-US)
+    '\x7F-\x9F'      # DEL and C1 controls
+    '\u200B-\u200F'  # Zero-width chars, LRM, RLM, etc.
+    '\u202A-\u202E'  # Bidi embedding controls
+    '\u2060-\u206F'  # Format chars (word joiner, etc.)
+    '\uFEFF'         # BOM / Zero Width NBSP
+    '\uFFF0-\uFFF8'  # Specials
+    ']+',
+    flags=re.UNICODE
+)
+
+# Surrogates and private use areas
+# Shouldn't appear in valid UTF-8, but strip just in case
+STRIP_PRIVATE_RE = re.compile(
+    '['
+    '\uD800-\uDFFF'         # Surrogates
+    '\uE000-\uF8FF'         # Private Use Area
+    '\uF900-\uFAFF'         # CJK Compatibility Ideographs (keep? strip for safety)
+    '\uFE10-\uFE1F'         # Vertical Forms
+    '\uFE20-\uFE2F'         # Combining Half Marks
+    '\U000F0000-\U000FFFFF' # Supplementary Private Use Area-A
+    '\U00100000-\U0010FFFF' # Supplementary Private Use Area-B
+    ']+',
+    flags=re.UNICODE
+)
+
+def strip_modifiers(text):
+    def process_characters(text):
+        result = []
+        i = 0
+        while i < len(text):
+            char = text[i]
+            category = unicodedata.category(char)
+            
+            if category.startswith(('L', 'N', 'P', 'S')):
+                result.append(char)
+                i += 1
+
+            elif category.startswith(('M', 'Sk', 'Cf')) or char in '\u200d\u200c':
+                i += 1
+
+            else:
+                result.append(char)
+                i += 1
+                
+        return ''.join(result)
+
+    if text == None: return None
+
+    for char in invalid_rendering:
+        text = text.replace(char, " ")
+    
+    stripped = process_characters(text)
+    stripped = re.sub(r'[\uFE00-\uFE0F]', '', stripped)
+    stripped = re.sub(r'[\U000E0100-\U000E01EF]', '', stripped, flags=re.UNICODE)
+    stripped = re.sub(r'[\U0001F3FB-\U0001F3FF]', '', stripped, flags=re.UNICODE)
+    stripped = re.sub(r'[\u200D\u200C]', '', stripped)
+    stripped = re.sub(r'\r\n?', '\n', stripped)
+    stripped = STRIP_CONTROL_RE.sub('', stripped)
+
+    return stripped.strip()
+
+def sanitize_name(name):
+    if name is None: return None
+    
+    # Convert to string and normalize to NFKC
+    # NFKC: Compatibility decomposition followed by canonical composition
+    # This handles: ① to 1, Ⅰ to I, etc., while keeping composed forms
+    name = str(name)
+    name = unicodedata.normalize('NFKC', name)
+    
+    # Build result using category-based filtering
+    result = []
+    for char in name:
+        cat = unicodedata.category(char)
+        cat_prefix = cat[0] if cat else 'C'
+        
+        # Allow letters (L*), numbers (N*), and punctuation (P*)
+        if cat_prefix in ('L', 'N', 'P'): result.append(char)
+        
+        # Allow space separator, normalize to regular space
+        elif cat == 'Zs': result.append(' ')
+
+        # Convert line/paragraph separators to space
+        elif cat in ('Zl', 'Zp'): result.append(' ')
+        
+        # Allow spacing combining marks (Mc) for Indic, Hebrew, etc.
+        elif cat == 'Mc': result.append(char)
+        
+        # Allow modifier letters (Lm) - e.g., ʰ, ʱ, ː
+        elif cat == 'Lm': result.append(char)
+        
+        # Strip everything else:
+        # - Mn (Nonspacing Mark): diacritics, combining marks (Zalgo)
+        # - Me (Enclosing Mark): enclosing combining marks
+        # - C* (Controls, Format, Surrogates, Private Use, Unassigned)
+        # - S* (Symbols: currency, math, modifiers, other)
+    
+    name = ''.join(result)
+    
+    # Additional block-based stripping for symbols that categories missed
+    name = STRIP_BLOCKS_RE.sub('', name)
+    name = STRIP_CONTROL_RE.sub('', name)
+    name = STRIP_PRIVATE_RE.sub('', name)
+    
+    # Collapse multiple whitespace characters
+    name = re.sub(r'\s+', ' ', name)
+    
+    # Strip leading/trailing whitespace
+    name = name.strip()
+    
+    return name
+
+def strip_micron(text):
+    text = re.sub(r'`[FB][0-9a-fA-F]{3}', '', text)
+    text = re.sub(r'`[FB]T[0-9a-fA-F]{6}', '', text)
+    text = re.sub(r'`[!*_=]', '', text)
+    text = re.sub(r'`f`b', '', text)
+    text = re.sub(r'`f', '', text)
+    text = re.sub(r'`b', '', text)
+    text = re.sub(r'`<', '', text)
+    text = re.sub(r'`>', '', text)
+    text = re.sub(r'`{', '', text)
+    return text
+
+def strip_escaped_micron(text):
+    text = re.sub(r'¦[FB][0-9a-fA-F]{3}', '', text)
+    text = re.sub(r'¦[FB]T[0-9a-fA-F]{6}', '', text)
+    text = re.sub(r'¦[!*_=]', '', text)
+    text = re.sub(r'¦f`b', '', text)
+    text = re.sub(r'¦f', '', text)
+    text = re.sub(r'¦b', '', text)
+    text = re.sub(r'¦<', '', text)
+    text = re.sub(r'¦>', '', text)
+    text = re.sub(r'¦{', '', text)
+    return text
+
+def unescape_micron(text):
+    text = re.sub(r'¦([FB][0-9a-fA-F]{3})', r'`\1', text)
+    text = re.sub(r'¦([FB]T[0-9a-fA-F]{6})', r'`\1', text)
+    text = re.sub(r'¦([!*_=])', r'`\1', text)
+    text = re.sub(r'¦(f`b)', r'`\1', text)
+    text = re.sub(r'¦(f)', r'`\1', text)
+    text = re.sub(r'¦(b)', r'`\1', text)
+    text = re.sub(r'¦(<)', r'`\1', text)
+    text = re.sub(r'¦(>)', r'`\1', text)
+    text = re.sub(r'¦({)', r'`\1', text)
+    return text
+
+def strip_non_formatting_tags(text):
+    text = re.sub(r'`<', '', text)
+    text = re.sub(r'`>', '', text)
+    text = re.sub(r'`{', '', text)
+    text = re.sub(r'`r', '', text)
+    text = re.sub(r'`c', '', text)
+    text = re.sub(r'`l', '', text)
+    return text

nomadnet/vendor/Scrollable.py

@@ -9,8 +9,11 @@
 # GNU General Public License for more details
 # http://www.gnu.org/licenses/gpl-3.0.txt
 
+import RNS
+import time
 import urwid
 from urwid.widget import BOX, FIXED, FLOW
+from collections import deque
 
 # Scroll actions
 SCROLL_LINE_UP        = 'line up'
@@ -50,6 +53,9 @@ class Scrollable(urwid.WidgetDecoration):
         self._old_cursor_coords = None
         self._rows_max_cached = 0
         self.force_forward_keypress = force_forward_keypress
+        self.force_cursor_update = False
+        self.anchor_cursor_update = False
+        self.cursor_update_forced_at = 0
         super().__init__(widget)
 
     def render(self, size, focus=False):
@@ -78,6 +84,10 @@ class Scrollable(urwid.WidgetDecoration):
 
         if canv_cols <= maxcol and canv_rows <= maxrow:
             # Canvas is small enough to fit without trimming
+            if self._trim_top != 0 or self._scroll_action is not None:
+                # Reset potentially stale _trim_top
+                self._trim_top = 0
+                self._scroll_action = None
             return canv
 
         self._adjust_trim_top(canv, size)
@@ -104,7 +114,11 @@ class Scrollable(urwid.WidgetDecoration):
             # Trimmed canvas contains the cursor, e.g. in an Edit widget
             self._forward_keypress = True
         else:
-            if canv_full.cursor is not None:
+            force_check  = self.force_cursor_update and (time.time()-self.cursor_update_forced_at < 0.25)
+            force_check |= self.anchor_cursor_update
+            if canv_full.cursor is not None or force_check:
+                self.force_cursor_update = False
+                self.anchor_cursor_update = False
                 # Full canvas contains the cursor, but scrolled out of view
                 self._forward_keypress = False
                 
@@ -129,7 +143,7 @@ class Scrollable(urwid.WidgetDecoration):
                                     first_visible = True
 
                                 if w.selectable():
-                                    ow.focus_item = pwi
+                                    ow.focus = pwi
 
                                     st = None
                                     nf = ow.get_focus()
@@ -346,6 +360,23 @@ class ScrollBar(urwid.WidgetDecoration):
         self.scrollbar_side = side
         self.scrollbar_width = max(1, width)
         self._original_widget_size = (0, 0)
+        self._sb_visible = False
+        self._sb_col_start = 0
+        self._sb_col_end = 0
+        self._sb_thumb_height = 1
+        self._sb_ow_rows_max = 0
+        self._sb_maxrow = 0
+        self._sb_dragging = False
+        self._sb_drag_offset = 0
+        self._accel = 1
+        self._sevt_interval = 0
+        self._sevt_start = 0
+        self._sfreq_deque = deque(maxlen=5)
+        
+        self._debug_hz = 0
+        self._debug_hz_smoothed = 0
+        self._debug_scroll_ms = 0
+        self._debug_scroll_started = 0
 
     def render(self, size, focus=False):
         maxcol, maxrow = size
@@ -360,15 +391,29 @@ class ScrollBar(urwid.WidgetDecoration):
         if ow_rows_max <= maxrow:
             # Canvas fits without scrolling - no scrollbar needed
             self._original_widget_size = size
+            self._sb_visible = False
             return ow.render(size, focus)
         ow_rows_max = ow_base.rows_max(ow_size, focus)
 
         ow_canv = ow.render(ow_size, focus)
+        
+        if self._debug_scroll_started:
+            total_ms = (time.time() - self._debug_scroll_started) * 1000
+            # RNS.log(f" accel={self._accel} , hz={RNS.prettyfrequency(self._debug_hz)} smoothed={RNS.prettyfrequency(self._debug_hz_smoothed)} scroll={self._debug_scroll_ms:.1f}ms total={total_ms:.1f}ms", RNS.LOG_DEBUG)
+            self._debug_scroll_started = 0
+        
         self._original_widget_size = ow_size
 
         pos = ow_base.get_scrollpos(ow_size, focus)
         posmax = ow_rows_max - maxrow
 
+        # Clamp position to valid scroll range. The position
+        # reported by the underlying widget can transiently
+        # fall outside [0, posmax] if its layout state is
+        # inconsistent.
+        if pos < 0: pos = 0
+        elif pos > posmax: pos = posmax
+
         # Thumb shrinks/grows according to the ratio of
         # <number of visible lines> / <number of total lines>
         thumb_weight = min(1, maxrow / max(1, ow_rows_max))
@@ -380,9 +425,12 @@ class ScrollBar(urwid.WidgetDecoration):
         if top_height == 0 and top_weight > 0:
             top_height = 1
 
-        # Bottom part is remaining space
+        # Bottom part is remaining space. Guard against
+        # negative values so the scrollbar canvas can
+        # never be taller than maxrow.
         bottom_height = maxrow - thumb_height - top_height
-        assert thumb_height + top_height + bottom_height == maxrow
+        if bottom_height < 0: bottom_height = 0
+        assert thumb_height + top_height + bottom_height <= maxrow
 
         # Create scrollbar canvas
         # Creating SolidCanvases of correct height may result in "cviews do not
@@ -400,9 +448,18 @@ class ScrollBar(urwid.WidgetDecoration):
 
         combinelist = [(ow_canv, None, True, ow_size[0]),
                        (sb_canv, None, False, sb_width)]
+
+        self._sb_visible = True
+        self._sb_thumb_height = thumb_height
+        self._sb_ow_rows_max = ow_rows_max
+        self._sb_maxrow = maxrow
         if self._scrollbar_side != SCROLLBAR_LEFT:
+            self._sb_col_start = ow_size[0]
+            self._sb_col_end = ow_size[0] + sb_width
             return urwid.CanvasJoin(combinelist)
         else:
+            self._sb_col_start = 0
+            self._sb_col_end = sb_width
             return urwid.CanvasJoin(reversed(combinelist))
 
     @property
@@ -450,21 +507,77 @@ class ScrollBar(urwid.WidgetDecoration):
     def mouse_event(self, size, event, button, col, row, focus):
         ow = self._original_widget
         ow_size = self._original_widget_size
+        ow_base = self.scrolling_base_widget if hasattr(self, 'scrolling_base_widget') else ow
+        on_sb = (self._sb_visible
+                 and self._sb_col_start <= col < self._sb_col_end)
+
+        if self._sb_dragging:
+            if 'drag' in event:
+                self._set_scrollpos_from_row(row)
+                return True
+            if 'release' in event or 'press' in event:
+                self._sb_dragging = False
+                if 'release' in event:
+                    return True
+
+        # Begin a click/drag
+        if on_sb and 'press' in event and button == 1 and hasattr(ow_base, 'set_scrollpos'):
+            pos = ow_base.get_scrollpos(ow_size)
+            posmax = max(1, self._sb_ow_rows_max - self._sb_maxrow)
+            track_len = max(1, self._sb_maxrow - self._sb_thumb_height)
+            top_height = int(round(track_len * pos / posmax))
+            if top_height <= row < top_height + self._sb_thumb_height:
+                self._sb_drag_offset = row - top_height
+            else:
+                # jump
+                self._sb_drag_offset = self._sb_thumb_height // 2
+                self._set_scrollpos_from_row(row)
+            self._sb_dragging = True
+            return True
+
         handled = False
         if hasattr(ow, 'mouse_event'):
             handled = ow.mouse_event(ow_size, event, button, col, row, focus)
 
         if not handled and hasattr(ow, 'set_scrollpos'):
-            if button == 4:    # scroll wheel up
+            ow.force_cursor_update = True
+            ow.cursor_update_forced_at = time.time()
+            self._sevt_interval = time.time()-self._sevt_start
+            if self._sevt_interval > 0.3: self._sevt_start = 0; self._accel = 1; self._sfreq_deque.clear()
+            if self._sevt_start == 0: self._sevt_start = time.time()
+            if self._sevt_interval > 0:
+                sevt_freq = 1/self._sevt_interval
+                self._sfreq_deque.append(sevt_freq)
+                freq_smth = sum(self._sfreq_deque)/len(self._sfreq_deque)
+                self._debug_hz          = sevt_freq
+                self._debug_hz_smoothed = freq_smth
+                if len(self._sfreq_deque) >= self._sfreq_deque.maxlen:
+                    if freq_smth > 25.0:  self._accel = max(self._accel, 3)
+                    elif freq_smth > 12.0: self._accel = max(self._accel, 2)
+                    else:                 self._accel = max(self._accel, 1)
+                
+            step = self._accel
+            if button in (4, 5):
+                scroll_started = time.time()
                 pos = ow.get_scrollpos(ow_size)
-                newpos = pos - 1
+                if button == 4: newpos = max(0, pos - step)
-                if newpos < 0:
+                else:           newpos = pos + step
-                    newpos = 0
                 ow.set_scrollpos(newpos)
-                return True
+                self._debug_scroll_started = scroll_started
-            elif button == 5:  # scroll wheel down
+                self._debug_scroll_ms      = (time.time() - scroll_started) * 1000
-                pos = ow.get_scrollpos(ow_size)
-                ow.set_scrollpos(pos + 1)
                 return True
 
         return False
+
+    def _set_scrollpos_from_row(self, row):
+        ow_base = self.scrolling_base_widget
+        if not hasattr(ow_base, 'set_scrollpos'):
+            return
+        posmax = self._sb_ow_rows_max - self._sb_maxrow
+        if posmax <= 0:
+            return
+        track_len = max(1, self._sb_maxrow - self._sb_thumb_height)
+        top = max(0, min(track_len, row - self._sb_drag_offset))
+        new_pos = int(round(top * posmax / track_len))
+        new_pos = max(0, min(posmax, new_pos))
+        ow_base.set_scrollpos(new_pos)

nomadnet/vendor/additional_urwid_widgets/FormWidgets.py

@@ -1,4 +1,13 @@
 import urwid
+import nomadnet
+
+from nomadnet.ui.textui.ReadlineEdit import ReadlineMixin, ReadlineEdit
+
+def dropdown_indicator():
+    try:
+        return nomadnet.NomadNetworkApp.get_shared_instance().ui.glyphs["dropdown"]
+    except Exception:
+        return " ▾"
 
 class DialogLineBox(urwid.LineBox):
     def __init__(self, body, parent=None, title="?"):
@@ -12,7 +21,7 @@ class DialogLineBox(urwid.LineBox):
             return None
         return super().keypress(size, key)
 
-class Placeholder(urwid.Edit):
+class Placeholder(ReadlineMixin, urwid.Edit):
     def __init__(self, caption="", edit_text="", placeholder="", **kwargs):
         super().__init__(caption, edit_text, **kwargs)
         self.placeholder = placeholder
@@ -36,6 +45,13 @@ class Dropdown(urwid.WidgetWrap):
         self.main_button = urwid.SelectableIcon(self.main_text, 0)
         self.main_button = urwid.AttrMap(self.main_button, "button_normal", "button_focus")
 
+        self.indicator_widget = urwid.Text(dropdown_indicator())
+        self.main_row = urwid.Columns([
+            ('pack', self.main_button),
+            ('pack', self.indicator_widget),
+            ('weight', 1, urwid.Text("")),
+        ])
+
         self.option_widgets = []
         for opt in options:
             icon = urwid.SelectableIcon(opt, 0)
@@ -46,7 +62,7 @@ class Dropdown(urwid.WidgetWrap):
         self.options_listbox = urwid.ListBox(self.options_walker)
         self.dropdown_box = None  # will be created on open_dropdown
 
-        self.pile = urwid.Pile([self.main_button])
+        self.pile = urwid.Pile([self.main_row])
         self.dropdown_visible = False
 
         super().__init__(self.pile)
@@ -123,9 +139,20 @@ class FormEdit(Placeholder, FormField):
                     self.error = "This field is required"
                     break
             elif validation == "number":
-                if value and not value.replace('-', '').replace('.', '').isdigit():
+                try:
-                    self.error = "This field must be a number"
+                    if value:
+                        int(value)
+                except ValueError:
+                    self.error = "This field must be a whole number"
                     break
+            elif validation == "stamp_cost":
+                if value and value.lower() != "none":
+                    try:
+                        if not 1 <= int(value) <= 255:
+                            raise ValueError
+                    except ValueError:
+                        self.error = "This field must be 1-255 or None"
+                        break
             elif validation == "float":
                 try:
                     if value:
@@ -156,7 +183,7 @@ class FormCheckbox(urwid.CheckBox, FormField):
         for validation in self.validation_types:
             if validation == "required":
                 if not value:
-                    self.error = "This field is  required"
+                    self.error = "This field is required"
                     break
 
         self.error_widget.set_text(("error", self.error or ""))
@@ -239,14 +266,25 @@ class FormMultiList(urwid.Pile, FormField):
         urwid.Pile.__init__(self, pile_widgets)
         FormField.__init__(self, config_key, transform)
 
-    def create_entry_row(self):
+    def create_entry_row(self, text=""):
-        edit = urwid.Edit("", "")
+        edit = ReadlineEdit("", text)
         entry_row = urwid.Columns([
             ('weight', 1, edit),
             (3, urwid.Button("×", on_press=lambda button: self.remove_entry(button, entry_row))),
         ])
         return entry_row
 
+    def set_value(self, value):
+        self.entries = []
+        if not value:
+            self.entries.append(self.create_entry_row())
+        else:
+            if isinstance(value, str):
+                value = [value]
+            for item in value:
+                self.entries.append(self.create_entry_row(str(item)))
+        self.contents = [(w, self.options()) for w in self.get_pile_widgets()]
+
     def remove_entry(self, button, entry_row):
         if len(self.entries) > 1:
             self.entries.remove(entry_row)
@@ -318,7 +356,7 @@ class FormMultiTable(urwid.Pile, FormField):
         if values is None:
             values = {}
 
-        name_edit = urwid.Edit("", name)
+        name_edit = ReadlineEdit("", name)
 
         columns = [('weight', 3, name_edit)]
 
@@ -330,9 +368,9 @@ class FormMultiTable(urwid.Pile, FormField):
                 widget = urwid.CheckBox("", state=bool(field_value))
             elif field_config.get("type") == "dropdown":
                 # TODO: dropdown in MultiTable
-                widget = urwid.Edit("", str(field_value))
+                widget = ReadlineEdit("", str(field_value))
             else:
-                widget = urwid.Edit("", str(field_value))
+                widget = ReadlineEdit("", str(field_value))
 
             field_widgets[field_key] = widget
             columns.append(('weight', 2, widget))
@@ -451,8 +489,8 @@ class FormKeyValuePairs(urwid.Pile, FormField):
         FormField.__init__(self, config_key, transform)
 
     def create_entry_row(self, key="", value=""):
-        key_edit = urwid.Edit("", key)
+        key_edit = ReadlineEdit("", key)
-        value_edit = urwid.Edit("", value)
+        value_edit = ReadlineEdit("", value)
 
         remove_button = urwid.Button("×", on_press=lambda button: self.remove_entry(button, entry_row))
 

nomadnet/vendor/additional_urwid_widgets/widgets/indicative_listbox.py

@@ -261,6 +261,11 @@ class IndicativeListBox(urwid.WidgetWrap):
             # position after the input is processed. If the list box body is empty, store None.
             focus_position_before_input = self.get_selected_position()
 
+            # Also remember the currently selected item itself. A button 1 press makes the contained list box move the focus, but
+            # the off focus highlighting is only (un)done during 'render()' while the widget has the focus. When the selection is
+            # changed by mouse while the widget is unfocused, the highlight would otherwise stay stuck on the previous item.
+            item_before_input = self.get_selected_item()
+
             # left mouse button, if not top bar or bottom bar.
             if (button == 1.0) and (topBar_rows <= row < (size[1] - bottomBar_rows)):
                 # Because 'row' includes the top bar, the offset must be substracted before passing it to the contained list box.
@@ -279,10 +284,19 @@ class IndicativeListBox(urwid.WidgetWrap):
                 
             focus_position_after_input = self.get_selected_position()
 
-            # If the focus position has changed, execute the hook (if existing).
+            if focus_position_before_input != focus_position_after_input:
-            if (focus_position_before_input != focus_position_after_input) and (self.on_selection_change is not None):
+                # The mouse moved the selection. If off focus highlighting is currently applied (the widget is unfocused), restore
-                self.on_selection_change(focus_position_before_input,
+                # the previously selected item and clear the stored state, so the next 'render()' re-applies the highlight to the
-                                         focus_position_after_input)
+                # newly selected item instead of leaving it stuck on the old one.
+                if (not self._last_focus_state) and (self._original_item_attr_map is not None) and (item_before_input is not None):
+                    item_before_input.set_attr_map(self._original_item_attr_map)
+                    self._original_item_attr_map = None
+                    self._last_focus_state = None
+
+                # If the focus position has changed, execute the hook (if existing).
+                if self.on_selection_change is not None:
+                    self.on_selection_change(focus_position_before_input,
+                                             focus_position_after_input)
         
         return was_handeled
     

nomadnet/vendor/cbor.py

@@ -0,0 +1,430 @@
+# https://github.com/brianolson/cbor_py
+# Copyright 2014-2015 Brian Olson
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import datetime
+import re
+import struct
+from io import BytesIO
+
+
+CBOR_TYPE_MASK = 0xE0  # top 3 bits
+CBOR_INFO_BITS = 0x1F  # low 5 bits
+
+
+CBOR_UINT    = 0x00
+CBOR_NEGINT  = 0x20
+CBOR_BYTES   = 0x40
+CBOR_TEXT    = 0x60
+CBOR_ARRAY   = 0x80
+CBOR_MAP     = 0xA0
+CBOR_TAG     = 0xC0
+CBOR_7       = 0xE0  # float and other types
+
+CBOR_UINT8_FOLLOWS  = 24  # 0x18
+CBOR_UINT16_FOLLOWS = 25  # 0x19
+CBOR_UINT32_FOLLOWS = 26  # 0x1a
+CBOR_UINT64_FOLLOWS = 27  # 0x1b
+CBOR_VAR_FOLLOWS    = 31  # 0x1f
+
+CBOR_BREAK  = 0xFF
+
+CBOR_FALSE  = (CBOR_7 | 20)
+CBOR_TRUE   = (CBOR_7 | 21)
+CBOR_NULL   = (CBOR_7 | 22)
+CBOR_UNDEFINED   = (CBOR_7 | 23)  # js 'undefined' value
+
+CBOR_FLOAT16 = (CBOR_7 | 25)
+CBOR_FLOAT32 = (CBOR_7 | 26)
+CBOR_FLOAT64 = (CBOR_7 | 27)
+
+CBOR_TAG_DATE_STRING = 0  # RFC3339
+CBOR_TAG_DATE_ARRAY = 1   # any number, seconds since 1970-01-01T00:00:00 UTC
+CBOR_TAG_BIGNUM = 2       # big-endian byte string follows
+CBOR_TAG_NEGBIGNUM = 3    # big-endian byte string follows
+CBOR_TAG_DECIMAL = 4
+CBOR_TAG_BIGFLOAT = 5
+CBOR_TAG_BASE64URL = 21
+CBOR_TAG_BASE64 = 22
+CBOR_TAG_BASE16 = 23
+CBOR_TAG_CBOR = 24
+
+CBOR_TAG_URI = 32
+CBOR_TAG_REGEX = 35
+CBOR_TAG_MIME = 36
+CBOR_TAG_CBOR_FILEHEADER = 55799  # 0xd9d9f7
+
+_CBOR_TAG_BIGNUM_BYTES = struct.pack('B', CBOR_TAG | CBOR_TAG_BIGNUM)
+_CBOR_TAG_NEGBIGNUM_BYTES = struct.pack('B', CBOR_TAG | CBOR_TAG_NEGBIGNUM)
+
+
+def _dumps_bignum_to_bytearray(val):
+    out = []
+    while val > 0:
+        out.insert(0, val & 0x0ff)
+        val = val >> 8
+    return bytes(out)
+
+
+def dumps_int(val):
+    "return bytes representing int val in CBOR"
+    if val >= 0:
+        if val <= 23:
+            return struct.pack('B', val)
+        if val <= 0x0ff:
+            return struct.pack('BB', CBOR_UINT8_FOLLOWS, val)
+        if val <= 0x0ffff:
+            return struct.pack('!BH', CBOR_UINT16_FOLLOWS, val)
+        if val <= 0x0ffffffff:
+            return struct.pack('!BI', CBOR_UINT32_FOLLOWS, val)
+        if val <= 0x0ffffffffffffffff:
+            return struct.pack('!BQ', CBOR_UINT64_FOLLOWS, val)
+        outb = _dumps_bignum_to_bytearray(val)
+        return _CBOR_TAG_BIGNUM_BYTES + _encode_type_num(CBOR_BYTES, len(outb)) + outb
+    val = -1 - val
+    return _encode_type_num(CBOR_NEGINT, val)
+
+
+def dumps_float(val):
+    return struct.pack("!Bd", CBOR_FLOAT64, val)
+
+
+def _encode_type_num(cbor_type, val):
+    """For some CBOR primary type [0..7] and an auxiliary unsigned number,
+    return CBOR encoded bytes."""
+    assert val >= 0
+    if val <= 23:
+        return struct.pack('B', cbor_type | val)
+    if val <= 0x0ff:
+        return struct.pack('BB', cbor_type | CBOR_UINT8_FOLLOWS, val)
+    if val <= 0x0ffff:
+        return struct.pack('!BH', cbor_type | CBOR_UINT16_FOLLOWS, val)
+    if val <= 0x0ffffffff:
+        return struct.pack('!BI', cbor_type | CBOR_UINT32_FOLLOWS, val)
+    if (((cbor_type == CBOR_NEGINT) and (val <= 0x07fffffffffffffff)) or
+        ((cbor_type != CBOR_NEGINT) and (val <= 0x0ffffffffffffffff))):
+        return struct.pack('!BQ', cbor_type | CBOR_UINT64_FOLLOWS, val)
+    if cbor_type != CBOR_NEGINT:
+        raise Exception("value too big for CBOR unsigned number: {0!r}".format(val))
+    outb = _dumps_bignum_to_bytearray(val)
+    return _CBOR_TAG_NEGBIGNUM_BYTES + _encode_type_num(CBOR_BYTES, len(outb)) + outb
+
+
+def dumps_string(val, is_text=None, is_bytes=None):
+    if isinstance(val, str):
+        val = val.encode('utf8')
+        is_text = True
+        is_bytes = False
+    if (is_bytes) or not (is_text == True):
+        return _encode_type_num(CBOR_BYTES, len(val)) + val
+    return _encode_type_num(CBOR_TEXT, len(val)) + val
+
+
+def dumps_array(arr, sort_keys=False):
+    head = _encode_type_num(CBOR_ARRAY, len(arr))
+    parts = [dumps(x, sort_keys=sort_keys) for x in arr]
+    return head + b''.join(parts)
+
+
+def dumps_dict(d, sort_keys=False):
+    head = _encode_type_num(CBOR_MAP, len(d))
+    parts = [head]
+    if sort_keys:
+        for k in sorted(d.keys()):
+            v = d[k]
+            parts.append(dumps(k, sort_keys=sort_keys))
+            parts.append(dumps(v, sort_keys=sort_keys))
+    else:
+        for k, v in d.items():
+            parts.append(dumps(k, sort_keys=sort_keys))
+            parts.append(dumps(v, sort_keys=sort_keys))
+    return b''.join(parts)
+
+
+def dumps_bool(b):
+    if b:
+        return struct.pack('B', CBOR_TRUE)
+    return struct.pack('B', CBOR_FALSE)
+
+
+def dumps_tag(t, sort_keys=False):
+    return _encode_type_num(CBOR_TAG, t.tag) + dumps(t.value, sort_keys=sort_keys)
+
+
+def dumps(ob, sort_keys=False):
+    if ob is None:
+        return struct.pack('B', CBOR_NULL)
+    if isinstance(ob, bool):
+        return dumps_bool(ob)
+    if isinstance(ob, (str, bytes, bytearray)):
+        if isinstance(ob, bytearray):
+            ob = bytes(ob)
+        return dumps_string(ob)
+    if isinstance(ob, (list, tuple)):
+        return dumps_array(ob, sort_keys=sort_keys)
+    if isinstance(ob, dict):
+        return dumps_dict(ob, sort_keys=sort_keys)
+    if isinstance(ob, float):
+        return dumps_float(ob)
+    if isinstance(ob, int):
+        return dumps_int(ob)
+    if isinstance(ob, Tag):
+        return dumps_tag(ob, sort_keys=sort_keys)
+    raise Exception("don't know how to cbor serialize object of type %s" % type(ob))
+
+
+def dump(obj, fp, sort_keys=False):
+    """obj: Python object to serialize. fp: file-like object capable of .write(bytes)."""
+    blob = dumps(obj, sort_keys=sort_keys)
+    fp.write(blob)
+
+
+class Tag(object):
+    def __init__(self, tag=None, value=None):
+        self.tag = tag
+        self.value = value
+
+    def __repr__(self):
+        return "Tag({0!r}, {1!r})".format(self.tag, self.value)
+
+    def __eq__(self, other):
+        if not isinstance(other, Tag):
+            return False
+        return (self.tag == other.tag) and (self.value == other.value)
+
+    def __hash__(self):
+        return hash((self.tag, self.value)) if not isinstance(self.value, (list, dict, bytearray)) else id(self)
+
+
+def loads(data):
+    """Parse CBOR bytes and return Python objects."""
+    if data is None:
+        raise ValueError("got None for buffer to decode in loads")
+    if isinstance(data, (bytes, bytearray, memoryview)):
+        fp = BytesIO(bytes(data))
+    else:
+        fp = data
+    return _loads(fp)[0]
+
+
+def load(fp):
+    """Parse and return object from fp, a file-like object supporting .read(n)."""
+    return _loads(fp)[0]
+
+
+_MAX_DEPTH = 100
+
+
+def _tag_aux(fp, tb):
+    bytes_read = 1
+    tag = tb & CBOR_TYPE_MASK
+    tag_aux = tb & CBOR_INFO_BITS
+    if tag_aux <= 23:
+        aux = tag_aux
+    elif tag_aux == CBOR_UINT8_FOLLOWS:
+        data = fp.read(1)
+        aux = struct.unpack_from("!B", data, 0)[0]
+        bytes_read += 1
+    elif tag_aux == CBOR_UINT16_FOLLOWS:
+        data = fp.read(2)
+        aux = struct.unpack_from("!H", data, 0)[0]
+        bytes_read += 2
+    elif tag_aux == CBOR_UINT32_FOLLOWS:
+        data = fp.read(4)
+        aux = struct.unpack_from("!I", data, 0)[0]
+        bytes_read += 4
+    elif tag_aux == CBOR_UINT64_FOLLOWS:
+        data = fp.read(8)
+        aux = struct.unpack_from("!Q", data, 0)[0]
+        bytes_read += 8
+    else:
+        assert tag_aux == CBOR_VAR_FOLLOWS, "bogus tag {0:02x}".format(tb)
+        aux = None
+
+    return tag, tag_aux, aux, bytes_read
+
+
+def _read_byte(fp):
+    tb = fp.read(1)
+    if len(tb) == 0:
+        raise EOFError()
+    return tb[0]
+
+
+def _loads_var_array(fp, limit, depth, returntags, bytes_read):
+    ob = []
+    tb = _read_byte(fp)
+    while tb != CBOR_BREAK:
+        (subob, sub_len) = _loads_tb(fp, tb, limit, depth, returntags)
+        bytes_read += 1 + sub_len
+        ob.append(subob)
+        tb = _read_byte(fp)
+    return (ob, bytes_read + 1)
+
+
+def _loads_var_map(fp, limit, depth, returntags, bytes_read):
+    ob = {}
+    tb = _read_byte(fp)
+    while tb != CBOR_BREAK:
+        (subk, sub_len) = _loads_tb(fp, tb, limit, depth, returntags)
+        bytes_read += 1 + sub_len
+        (subv, sub_len) = _loads(fp, limit, depth, returntags)
+        bytes_read += sub_len
+        ob[subk] = subv
+        tb = _read_byte(fp)
+    return (ob, bytes_read + 1)
+
+
+def _loads_array(fp, limit, depth, returntags, aux, bytes_read):
+    ob = []
+    for _ in range(aux):
+        subob, subpos = _loads(fp, limit, depth, returntags)
+        bytes_read += subpos
+        ob.append(subob)
+    return ob, bytes_read
+
+
+def _loads_map(fp, limit, depth, returntags, aux, bytes_read):
+    ob = {}
+    for _ in range(aux):
+        subk, subpos = _loads(fp, limit, depth, returntags)
+        bytes_read += subpos
+        subv, subpos = _loads(fp, limit, depth, returntags)
+        bytes_read += subpos
+        ob[subk] = subv
+    return ob, bytes_read
+
+
+def _loads(fp, limit=None, depth=0, returntags=False):
+    "return (object, bytes read)"
+    if depth > _MAX_DEPTH:
+        raise Exception("hit CBOR loads recursion depth limit")
+    tb = _read_byte(fp)
+    return _loads_tb(fp, tb, limit, depth, returntags)
+
+
+def _loads_tb(fp, tb, limit=None, depth=0, returntags=False):
+    # Some special cases of CBOR_7 best handled by special struct.unpack logic here
+    if tb == CBOR_FLOAT16:
+        data = fp.read(2)
+        hibyte, lowbyte = struct.unpack_from("BB", data, 0)
+        exp = (hibyte >> 2) & 0x1F
+        mant = ((hibyte & 0x03) << 8) | lowbyte
+        if exp == 0:
+            val = mant * (2.0 ** -24)
+        elif exp == 31:
+            val = float('Inf') if mant == 0 else float('NaN')
+        else:
+            val = (mant + 1024.0) * (2 ** (exp - 25))
+        if hibyte & 0x80:
+            val = -1.0 * val
+        return (val, 3)
+    elif tb == CBOR_FLOAT32:
+        data = fp.read(4)
+        pf = struct.unpack_from("!f", data, 0)
+        return (pf[0], 5)
+    elif tb == CBOR_FLOAT64:
+        data = fp.read(8)
+        pf = struct.unpack_from("!d", data, 0)
+        return (pf[0], 9)
+
+    tag, tag_aux, aux, bytes_read = _tag_aux(fp, tb)
+
+    if tag == CBOR_UINT:
+        return (aux, bytes_read)
+    elif tag == CBOR_NEGINT:
+        return (-1 - aux, bytes_read)
+    elif tag == CBOR_BYTES:
+        ob, subpos = loads_bytes(fp, aux)
+        return (ob, bytes_read + subpos)
+    elif tag == CBOR_TEXT:
+        raw, subpos = loads_bytes(fp, aux, btag=CBOR_TEXT)
+        ob = raw.decode('utf8')
+        return (ob, bytes_read + subpos)
+    elif tag == CBOR_ARRAY:
+        if aux is None:
+            return _loads_var_array(fp, limit, depth, returntags, bytes_read)
+        return _loads_array(fp, limit, depth, returntags, aux, bytes_read)
+    elif tag == CBOR_MAP:
+        if aux is None:
+            return _loads_var_map(fp, limit, depth, returntags, bytes_read)
+        return _loads_map(fp, limit, depth, returntags, aux, bytes_read)
+    elif tag == CBOR_TAG:
+        ob, subpos = _loads(fp, limit, depth + 1, returntags)
+        bytes_read += subpos
+        if returntags:
+            ob = Tag(aux, ob)
+        else:
+            ob = tagify(ob, aux)
+        return ob, bytes_read
+    elif tag == CBOR_7:
+        if tb == CBOR_TRUE:
+            return (True, bytes_read)
+        if tb == CBOR_FALSE:
+            return (False, bytes_read)
+        if tb == CBOR_NULL:
+            return (None, bytes_read)
+        if tb == CBOR_UNDEFINED:
+            return (None, bytes_read)
+        raise ValueError("unknown cbor tag 7 byte: {:02x}".format(tb))
+
+
+def loads_bytes(fp, aux, btag=CBOR_BYTES):
+    if aux is not None:
+        ob = fp.read(aux)
+        return (ob, aux)
+    chunklist = []
+    total_bytes_read = 0
+    while True:
+        tb = fp.read(1)[0]
+        if tb == CBOR_BREAK:
+            total_bytes_read += 1
+            break
+        tag, tag_aux, aux, bytes_read = _tag_aux(fp, tb)
+        assert tag == btag, 'variable length value contains unexpected component'
+        ob = fp.read(aux)
+        chunklist.append(ob)
+        total_bytes_read += bytes_read + aux
+    return (b''.join(chunklist), total_bytes_read)
+
+
+def _bytes_to_biguint(bs):
+    out = 0
+    for ch in bs:
+        out = out << 8
+        out = out | ch
+    return out
+
+
+def tagify(ob, aux):
+    if aux == CBOR_TAG_DATE_STRING:
+        # RFC3339 date string parsing not implemented; return as Tag.
+        return Tag(aux, ob)
+    if aux == CBOR_TAG_DATE_ARRAY:
+        return datetime.datetime.fromtimestamp(ob, tz=datetime.timezone.utc)
+    if aux == CBOR_TAG_BIGNUM:
+        return _bytes_to_biguint(ob)
+    if aux == CBOR_TAG_NEGBIGNUM:
+        return -1 - _bytes_to_biguint(ob)
+    if aux == CBOR_TAG_REGEX:
+        return re.compile(ob)
+    return Tag(aux, ob)
+
+
+def encode(obj, sort_keys=False):
+    return dumps(obj, sort_keys=sort_keys)
+
+
+def decode(data):
+    return loads(data)

setup.py

@@ -1,3 +1,4 @@
+import sys
 import setuptools
 
 exec(open("nomadnet/_version.py", "r").read())
@@ -5,9 +6,14 @@ exec(open("nomadnet/_version.py", "r").read())
 with open("README.md", "r") as fh:
     long_description = fh.read()
 
+if "--getversion" in sys.argv:
+    print(__version__, end="")
+    exit(0)
+
 package_data = {
 "": [
     "examples/messageboard/*",
+    "ui/textui/images/demo.webp",
     ]
 }
 
@@ -30,6 +36,6 @@ setuptools.setup(
     entry_points= {
         'console_scripts': ['nomadnet=nomadnet.nomadnet:main']
     },
-    install_requires=["rns>=1.0.4", "lxmf>=0.9.3", "urwid>=2.6.16", "qrcode"],
+    install_requires=["rns>=1.5.5", "lxmf>=1.2.0", "urwid>=3.0.5", "qrcode"],
-    python_requires=">=3.7",
+    python_requires=">=3.8",
 )

tools/colorgen.py

@@ -0,0 +1,321 @@
+#!/usr/bin/env python3
+"""
+Color Table Generator with Perceptual Normalization
+
+Generates HSL color tables with optional normalization to uniform perceptual
+intensity (CIELAB L*) followed by global lightness scaling.
+"""
+
+import argparse
+import colorsys
+import math
+import sys
+
+
+def hsl_to_rgb(h, s, l):
+    """Convert HSL (0-360, 0-100, 0-100) to RGB (0.0-1.0)."""
+    h_norm = (h % 360) / 360.0
+    s_norm = max(0, min(100, s)) / 100.0
+    l_norm = max(0, min(100, l)) / 100.0
+    r, g, b = colorsys.hls_to_rgb(h_norm, l_norm, s_norm)
+    return (r, g, b)
+
+
+def rgb_to_hex(r, g, b):
+    """Convert RGB 0.0-1.0 to lowercase hex string."""
+    r = max(0, min(1, r))
+    g = max(0, min(1, g))
+    b = max(0, min(1, b))
+    return f"{int(r*255):02x}{int(g*255):02x}{int(b*255):02x}"
+
+
+# CIELAB conversion functions
+def srgb_to_linear(c):
+    if c <= 0.04045:
+        return c / 12.92
+    return ((c + 0.055) / 1.055) ** 2.4
+
+
+def linear_to_srgb(c):
+    if c <= 0.0031308:
+        return c * 12.92
+    return 1.055 * (c ** (1/2.4)) - 0.055
+
+
+def rgb_to_lab(r, g, b):
+    """Convert sRGB (0-1) to CIELAB (L: 0-100, a,b: ~-128 to 127). D65 illuminant."""
+    # sRGB to Linear
+    r_lin = srgb_to_linear(r)
+    g_lin = srgb_to_linear(g)
+    b_lin = srgb_to_linear(b)
+    
+    # Linear to XYZ (D65)
+    X = 0.4124564 * r_lin + 0.3575761 * g_lin + 0.1804375 * b_lin
+    Y = 0.2126729 * r_lin + 0.7151522 * g_lin + 0.0721750 * b_lin
+    Z = 0.0193339 * r_lin + 0.1191920 * g_lin + 0.9503041 * b_lin
+    
+    # XYZ to LAB
+    Xn, Yn, Zn = 95.047, 100.0, 108.883
+    delta = 6/29
+    
+    def f(t):
+        if t > delta**3:
+            return t ** (1/3)
+        return t / (3 * delta**2) + 4/29
+    
+    L = 116 * f(Y / Yn) - 16
+    a = 500 * (f(X / Xn) - f(Y / Yn))
+    b_val = 200 * (f(Y / Yn) - f(Z / Zn))
+    
+    return (L, a, b_val)
+
+
+def lab_to_rgb(L, a, b_val):
+    """Convert CIELAB to sRGB (0-1)."""
+    Xn, Yn, Zn = 95.047, 100.0, 108.883
+    delta = 6/29
+    
+    def inv_f(y):
+        if y > delta:
+            return y ** 3
+        return 3 * delta**2 * (y - 4/29)
+    
+    L_adj = (L + 16) / 116
+    X = Xn * inv_f(L_adj + a / 500)
+    Y = Yn * inv_f(L_adj)
+    Z = Zn * inv_f(L_adj - b_val / 200)
+    
+    # XYZ to Linear RGB
+    r_lin =  3.2404542 * X - 1.5371385 * Y - 0.4985314 * Z
+    g_lin = -0.9692660 * X + 1.8760108 * Y + 0.0415560 * Z
+    b_lin =  0.0556434 * X - 0.2040259 * Y + 1.0572252 * Z
+    
+    r = linear_to_srgb(r_lin)
+    g = linear_to_srgb(g_lin)
+    b = linear_to_srgb(b_lin)
+    
+    return (r, g, b)
+
+
+def generate_colors(
+    hues=None,
+    hue_start=0.0,
+    hue_step=30.0,
+    sat_start=70.0,
+    sat_step=0.0,
+    sat_steps=1,
+    light_start=50.0,
+    light_step=0.0,
+    light_steps=1,
+    perceptual_multiplier=1.0,
+    use_hsl_space=False,
+    normalize=False,
+    normalize_target=None,
+    discards=[]
+):
+    """
+    Generate color table with optional perceptual normalization.
+    
+    Pipeline:
+    1. Generate HSL variations
+    2. If normalize: Convert to LAB, set all L* to target (mean or specified)
+    3. Apply global perceptual multiplier to L*
+    4. Convert to RGB and output hex
+    """
+    # Determine hue list
+    if hues is not None:
+        hue_list = [float(h) % 360 for h in hues]
+    else:
+        hue_list = []
+        current = hue_start % 360
+        if hue_step <= 0:
+            raise ValueError("hue_step must be positive when using start/step mode")
+        while current < 360:
+            hue_list.append(current)
+            current += hue_step
+    
+    # Determine if we need LAB processing
+    needs_lab = normalize or (perceptual_multiplier != 1.0) or not use_hsl_space
+    
+    colors = []
+    lab_colors = []  # Store as (L, a, b) tuples if processing needed
+    
+    for h in hue_list:
+        for i in range(sat_steps):
+            s = sat_start + (i * sat_step)
+            s = max(0, min(100, s))
+            
+            for j in range(light_steps):
+                # l = light_start + (j * light_step)
+                l = light_start + (i * light_step)
+                l = max(0, min(100, l))
+                
+                r, g, b = hsl_to_rgb(h, s, l)
+                
+                if not needs_lab:
+                    colors.append(rgb_to_hex(r, g, b))
+                else:
+                    L, a, b_lab = rgb_to_lab(r, g, b)
+                    lab_colors.append((L, a, b_lab))
+
+    filtered_colors = []
+    filtered_lab_colors = []
+    i = 0
+    for c in colors:
+        i += 1
+        if not str(i) in discards:
+            filtered_colors.append(c)
+
+    i = 0
+    for c in lab_colors:
+        i += 1
+        if not str(i) in discards:
+            filtered_lab_colors.append(c)
+
+    colors = filtered_colors
+    lab_colors = filtered_lab_colors
+
+    if not needs_lab:
+        return colors
+
+    # Step 2: Perceptual Normalization (equalize intensity)
+    if normalize:
+        if normalize_target is not None:
+            target_L = normalize_target
+        else:
+            # Calculate mean perceptual lightness
+            target_L = sum(lab[0] for lab in lab_colors) / len(lab_colors)
+            print(f"Target L*: {target_L}")
+        
+        # Clamp target to valid LAB range
+        target_L = max(0, min(100, target_L))
+        
+        # Normalize all colors to target L, preserving hue (a, b)
+        lab_colors = [(target_L, a, b) for (L, a, b) in lab_colors]
+    
+    # Step 3: Apply final global perceptual multiplier
+    final_colors = []
+    for L, a, b in lab_colors:
+        L_final = L * perceptual_multiplier
+        L_final = max(0, min(100, L_final))
+        r, g, b = lab_to_rgb(L_final, a, b)
+        final_colors.append(rgb_to_hex(r, g, b))
+    
+    return final_colors
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Generate perceptually normalized color tables",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog="""
+Pipeline: HSL Generation → [Normalize to uniform L*] → Global Multiplier → Hex
+
+Examples:
+  # 32 colors normalized to same perceptual intensity, then darkened 20%
+  python3 colorgen.py --hue-start 0 --hue-step 11.25 --sat-start 75 --normalize --perceptual-multiplier 0.8
+  
+  # Force all colors to exactly L*=55, then scale by 1.1
+  python3 colorgen.py --hues "0,90,180,270" --normalize --normalize-target 55 --perceptual-multiplier 1.1
+  
+  # Generate with lightness variation, normalize to mean intensity, output brightened
+  python3 colorgen.py --hue-start 15 --hue-step 30 --light-start 40 --light-step 10 --light-steps 3 --normalize --perceptual-multiplier 1.2
+        """
+    )
+    
+    # Hue configuration
+    hue_group = parser.add_mutually_exclusive_group()
+    hue_group.add_argument("--hues", type=str, metavar="LIST",
+                          help='Comma-separated hue values, e.g., "0,60,120"')
+    hue_group.add_argument("--hue-start", type=float, default=0.0,
+                          help="Initial hue offset in degrees")
+    parser.add_argument("--hue-step", type=float, default=30.0,
+                       help="Separation between hues")
+    
+    # Saturation/Lightness configuration
+    parser.add_argument("--sat-start", type=float, default=75.0,
+                       help="Initial saturation (0-100)")
+    parser.add_argument("--sat-step", type=float, default=0.0,
+                       help="Saturation step")
+    parser.add_argument("--sat-steps", type=int, default=1,
+                       help="Number of saturation variations")
+    parser.add_argument("--light-start", type=float, default=50.0,
+                       help="Initial lightness (0-100)")
+    parser.add_argument("--light-step", type=float, default=0.0,
+                       help="Lightness step")
+    parser.add_argument("--light-steps", type=int, default=1,
+                       help="Number of lightness variations")
+    
+    # Perceptual processing
+    parser.add_argument("--normalize", action="store_true",
+                       help="Normalize all colors to same perceptual intensity (L*) before applying multiplier")
+    parser.add_argument("--normalize-target", type=float, default=None, metavar="L",
+                       help="Target L* value (0-100) for normalization. Default: mean of generated colors")
+    parser.add_argument("--perceptual-multiplier", type=float, default=1.0,
+                       help="Final global lightness multiplier (1.0=no change)")
+    parser.add_argument("--hsl-space", action="store_true",
+                       help="Skip LAB conversion if not normalizing (faster, less accurate perceptually)")
+    
+    # Output
+    parser.add_argument("--python-list", action="store_true",
+                       help="Output as Python list")
+    parser.add_argument("--separator", type=str, default="\n",
+                       help="Separator between colors")
+    parser.add_argument("--stats", action="store_true",
+                       help="Print generation stats to stderr")
+    hue_group.add_argument("--discard", type=str, metavar="LIST",
+                          help='Discard output indexes, e.g., "20,2,33"')
+    
+    args = parser.parse_args()
+    
+    if args.hsl_space and args.normalize:
+        print("Warning: --hsl-space ignored because --normalize requires LAB space", file=sys.stderr)
+        args.hsl_space = False
+    
+    try:
+        if args.discard: discards = args.discard.split(",")
+        else: discards = []
+
+        colors = generate_colors(
+            hues=args.hues.split(",") if args.hues else None,
+            hue_start=args.hue_start,
+            hue_step=args.hue_step,
+            sat_start=args.sat_start,
+            sat_step=args.sat_step,
+            sat_steps=args.sat_steps,
+            light_start=args.light_start,
+            light_step=args.light_step,
+            light_steps=args.light_steps,
+            perceptual_multiplier=args.perceptual_multiplier,
+            use_hsl_space=args.hsl_space,
+            normalize=args.normalize,
+            normalize_target=args.normalize_target,
+            discards=discards,
+        )
+        
+        i = 0
+        # Enable to test on light background
+        # print(f"#!bg=fff\n")
+        if args.python_list: print("colors = [", end="")
+        for c in colors:
+            i+=1
+            if args.python_list:
+                print(f"\"{c}\", ", end="")
+
+            else: print(f"`FT{c}colored {i}`f")
+        
+        if args.python_list: print("]")
+
+        if args.stats:
+            total = len(colors)
+            mode = "normalized + " if args.normalize else ""
+            print(f"# Generated {total} colors ({mode}L* × {args.perceptual_multiplier})", 
+                  file=sys.stderr)
+                  
+    except Exception as e:
+        print(f"Error: {e}", file=sys.stderr)
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

tools/rrc-log-dump.py

@@ -0,0 +1,118 @@
+#!/usr/bin/env python3
+"""Dump the contents of rrc on-disk room history logs.
+
+Usage:
+    rrc-log-dump.py <path>
+
+<path> may be:
+  - a single .log file
+  - a hub directory under rrc_history/
+  - the rrc_history/ root (or any ancestor; .log files are found recursively)
+"""
+
+import os
+import sys
+import time
+
+SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.join(SCRIPT_DIR, "..", "nomadnet", "vendor"))
+import cbor
+
+
+H_KIND    = "k"
+H_SRC     = "s"
+H_NICK    = "n"
+H_TEXT    = "t"
+H_TS      = "ts"
+H_MENTION = "m"
+
+KIND_COLOR = {
+    "msg":    "",
+    "system": "\033[2m",
+    "notice": "\033[36m",
+    "error":  "\033[31m",
+}
+RESET = "\033[0m"
+
+
+def _fmt_ts(ts_ms):
+    try:
+        return time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(int(ts_ms) // 1000))
+    except Exception:
+        return "????-??-?? ??:??:??"
+
+
+def _fmt_speaker(entry):
+    nick = entry.get(H_NICK)
+    src  = entry.get(H_SRC)
+    src_hex = src.hex()[:12] if isinstance(src, (bytes, bytearray)) else None
+    if nick and src_hex:
+        return "%s (%s)" % (nick, src_hex)
+    if nick:
+        return nick
+    if src_hex:
+        return src_hex
+    return ""
+
+
+def dump_file(path, use_color):
+    print("=== %s ===" % path)
+    count = 0
+    truncated = False
+    with open(path, "rb") as f:
+        while True:
+            try:
+                entry = cbor.load(f)
+            except EOFError:
+                break
+            except Exception as ex:
+                truncated = True
+                print("  [corrupt record after entry %d: %s]" % (count, ex), file=sys.stderr)
+                break
+            if not isinstance(entry, dict):
+                print("  [skipping non-dict entry: %r]" % (entry,), file=sys.stderr)
+                continue
+            kind    = entry.get(H_KIND) or "msg"
+            ts      = _fmt_ts(entry.get(H_TS, 0))
+            speaker = _fmt_speaker(entry)
+            text    = entry.get(H_TEXT) or ""
+            mention = " *" if entry.get(H_MENTION) else "  "
+            color   = KIND_COLOR.get(kind, "") if use_color else ""
+            end     = RESET if (color and use_color) else ""
+            line = "%s  %-6s%s  %-32s  %s" % (ts, kind, mention, speaker, text)
+            print(color + line + end)
+            count += 1
+    suffix = " (truncated)" if truncated else ""
+    print("--- %d entries%s ---" % (count, suffix))
+
+
+def main(argv):
+    if len(argv) != 2 or argv[1] in ("-h", "--help"):
+        print(__doc__, file=sys.stderr)
+        return 2
+    target = argv[1]
+    use_color = sys.stdout.isatty()
+    if os.path.isfile(target):
+        dump_file(target, use_color)
+        return 0
+    if os.path.isdir(target):
+        files = []
+        for root, _dirs, names in os.walk(target):
+            for name in names:
+                if name.endswith(".log"):
+                    files.append(os.path.join(root, name))
+        files.sort()
+        if not files:
+            print("no .log files found under %s" % target, file=sys.stderr)
+            return 1
+        for i, path in enumerate(files):
+            if i > 0:
+                print()
+            dump_file(path, use_color)
+        return 0
+    print("not a file or directory: %s" % target, file=sys.stderr)
+    return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv))