wrolpi/modules/map/export_job.py

322 lines
14 KiB
Python
Raw Normal View History

A map is exported as a PDF: a 1:24,000 quadrangle, in layers, on a sheet, which says where it is (#1006) * The map can choose a 1:24,000 quadrangle to export (#971) The first part of exporting a map as a PDF: a grid of 7.5-minute quadrangles is drawn over the map, and a click chooses one. The grid is arithmetic, so it needs no data. Exporting itself is not here yet; the button is disabled. Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * A quadrangle of the map is exported as a PDF with a layer for each map layer (#972) * A quadrangle of the map is exported as a PDF with a layer for each map layer The browser draws the quadrangle off the screen at the size of a 1:24,000 sheet, once for each layer which is on, and the API puts the images in a PDF, each in a layer of its own which a reader can show and hide. A layer in which nothing is drawn is left out. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * The exported map's canvas keeps what was drawn, and an export's memory is bounded - The drawing buffer is asked for where this MapLibre reads it (canvasContextAttributes); as an option of the map it was ignored, so the canvas could be read after it was cleared. - Each layer's pixels are compressed in the PDF as the layer is added. PyMuPDF kept them uncompressed until the PDF was written: about 90 MB a layer of a quadrangle, all at once. - The pixels of all the layers together are limited, before any image is decoded. - pymupdf is imported when a PDF is made, not when the API starts. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * An exported map says where on the Earth it is (#973) * An exported map says where on the Earth it is The PDF's page has a geospatial viewport (ISO 32000-2): the page is the map in Web Mercator, with the latitude and longitude of each corner. The corners are the edges of the picture the browser drew, read from the map which drew it. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * A map whose edge is the antimeridian can be exported The map's arithmetic puts the east edge of the last quadrangle before the antimeridian a rounding error past 180 degrees, and the export was refused. An edge within 1e-8 of a degree of 180 is at 180 (it is not wrapped to -180); a map which is really across the antimeridian is still refused. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * An exported map is on a sheet: a margin, a grid, scale bars and notes (#974) * An exported map is on a sheet: a margin, a grid, scale bars and notes The map is put on a larger page with what a topographic map has around it: a title, the latitude and longitude of its corners and of ticks between them, scale bars (kilometers, miles, feet), and notes of the projection, the grid and where the map's data is from. Over the map is a 1,000 meter UTM grid, each line named in the margin. The grid and the margin are PDF layers of their own. The PDF says where the map is, not where the page is. The picker has a box to export the map alone. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * The scale of a narrow map is written under its name, not over it A quadrangle near a pole is two inches wide, which is not room for its name and its scale on one line. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * An exported map says where a compass points (#975) * An exported map says where a compass points The sheet has a diagram of the three norths (true, grid and magnetic) beside its scale bars, and a note of how far magnetic north is from true north on the day the map was made, and how fast it moves. The declination is from the World Magnetic Model 2025 (NOAA NCEI and the British Geological Survey; public domain), whose coefficient file is shipped: nothing is downloaded. The model is good until 2030; outside its years nothing is said of a compass. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * A corner of an exported map is not also a tick The edges of the picture the browser draws are a hair outside the quadrangle's, so the quadrangle's own edges were taken for ticks inside the map, and each was written over the coordinate of its corner. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * A compass which passes due south has moved a little, not most of the way around Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * A map can be exported at 400 DPI (#976) The picker has a choice of 300 or 400 DPI. A map is drawn less sharp than was asked for, rather than refused, when it would be more pixels than the API makes a PDF of: a quadrangle near the equator, or one with many layers on. The picker says so while it draws. Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * An exported quadrangle goes by the name the USGS gives it (#977) * An exported quadrangle goes by the name the USGS gives it The picker says what the chosen quadrangle is called, and the sheet's title, the PDF's title and the file's name have it. Where the USGS has no name, which is most of the world, a quadrangle goes by where it is, as before. The names are an index of the 65,600 7.5 minute quadrangles of The National Map (U.S. Geological Survey; public domain), which is shipped: nothing is downloaded. In Alaska a USGS quadrangle is wider than 7.5 minutes, so a quadrangle of the picker's grid there is "Part of" the one which has its middle. The places the tests of the map export are of are changed for others. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * An area half of which is the last quadrangle of the land is a part of it The middle of such an area is on the quadrangle's east or north edge, with no quadrangle beyond it, and it was given no name. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * The PDF of a map is made by a Job, which the browser watches (#978) * The PDF of a map is made by a Job, which the browser watches A PDF of a quadrangle is a minute or two of work on a Raspberry Pi 4, and the request which waited for it was cut off after a minute: only the smallest quadrangles could be exported there. Now the browser describes the map, sends the image of each layer in a request of its own, and starts a Job; it watches the Job, shows how far along it is, and fetches the PDF when the Job is done. Cancel stops the Job. The images are kept on disk until the PDF is made of them, and an export is removed when it is an hour old. The credits on the sheet are no longer cut off: an attribution which names many sources is each of them. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * A cancelled map export stops, and is removed - A cancelled Job is not over until the thread making the PDF has stopped, so the next Job does not start beside it; the work can also be stopped while the sheet is drawn and the PDF written. - DELETE /api/map/export/<id> stops an export at any step and removes what was sent; the browser calls it when the person exporting cancels, while layers are sent as well as while the PDF is made. - An export whose Job is waiting or running is not removed for being old; one whose Job was cancelled is removed at once. - Cancel while the map is drawn stops the drawing between two layers. - Exports are kept in a directory made for this user alone, which is refused when it is a link or is another user's; only what is an export there is removed. - A request for the finished PDF which is lost is made again. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * An overlay's areas and lines go under the map's labels, and its own labels over them (#979) An overlay was drawn whole over the basemap, so an area it filled covered the names of what is in it. Labels are now always on top: the areas, lines and pictures of every overlay are under the basemap's labels, and the text, icons and points of every overlay are over them. It is so in the map, the standalone viewer, and an exported PDF, where an overlay with both is two layers ("Towers" and "Towers (labels)"). Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * An overlay's ground is over the ground of every basemap file (#980) * An overlay's ground is over the ground of every basemap file On a map of several basemap files, the ground of an overlay was put before the first label of the first file, which is under the land, water and roads of every file after it: an overlay's areas and lines were hidden, and in an exported PDF its layer was just above the background. It goes where the basemap's labels begin: the first of the labels which are over all of the basemap's ground. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * The labels of every basemap file are over the ground of every file A map of several basemap files was their layers one file after another, so only the labels of the last file were on top: the labels of the others were under the land, water and roads of the files after them, and under the ground of an overlay. Both viewers now draw the ground of every file and then the labels of every file. The contour lines are drawn after the basemap's labels and are not of the basemap: an overlay's ground goes under the basemap's labels, not under the contours instead. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * Contours are drawn from as deep as the terrain file goes (#981) The map read a terrain file to zoom 8, which is how deep the terrain WROLPi offers goes, however deep the file was. It now reads a file as deep as the file says it goes, in both viewers, and draws finer contour lines from a deeper file: every 40 feet (or 20 meters) from zoom 11, as a 1:24,000 sheet has. A file which goes to zoom 8 is drawn as it was. Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * An exported map names its small roads (#982) * An exported map names its small roads A quadrangle is drawn at about zoom 13. The basemap's style names small roads from zoom 15, and its tiles have no names for them below zoom 14, so the streets of a town were printed as lines with no names. Before the map is drawn, the tiles of zoom 14 which the quadrangle is in are read, and the roads in them which have a name are given to the export map as a source of its own. The layer which names small roads draws from that source, as its style says to. At the scale of a sheet most streets are shorter than their names, and a map writes a name along a line only where the line is longer. Such a road is named over its middle, turned as the road goes; a road in pieces is named once. A source whose tiles can not be read is exported as before. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * The names of small roads are placed by the export, at every latitude A quadrangle nearer the equator is drawn at a little over zoom 14, where the names were not read and the style still kept them for zoom 15: such a map had no names of small roads. They are now read wherever a layer would not name roads. Every name of a small road is now a point: over the middle of its road, turned as the road goes under it, again every 400 pixels of a long road. Before, a road long enough was left for the map to name along its line, which the map often does not (the line bends, or is in pieces), and a short piece beside it was named too. One place decides where a road is named, so a road in pieces is named once, and a piece of another kind of road does not take the name. Reading the names can be cancelled. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * A sheet has a bar of meters, its norths in mils, and how far apart its contour lines are (#983) Under the map are bars of kilometers, meters, miles and feet. How far the grid's north and a compass's north are from true north is written in mils as well as in degrees, in the diagram and in the notes: a compass may be marked in either. A map which is exported with its contour lines says how far apart they are (CONTOUR INTERVAL 40 FEET), under the bars. The browser knows: it is the interval of the tiles the lines are made for at the zoom of the sheet, in the unit the map measures in. It says so with the sheet, and what is not an interval is not written. Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * A sheet says what the quadrangles around it are called (#984) * A sheet says what the quadrangles around it are called Under the scale bars is a diagram of nine squares, the map the middle one, and beside it what the USGS calls each of the eight around it: the sheets to print to carry on off the edge of this one. The names are from the index of quadrangles which names the sheet itself. A quadrangle with no name (out to sea, or over a border) has no number; a map of where the USGS names nothing, and a map too narrow for the diagram, go without. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * A sheet is never beside itself Where a quadrangle of the USGS is wider than the map (Alaska), the area the size of the map to its east or west is often more of the quadrangle the map is in, and the diagram named the sheet as its own neighbor. The quadrangle on a side is now the next one which is not the map's own. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * A sheet has a legend of the lines of its map (#985) * A sheet has a legend of the lines of its map Under the scale bars is what each kind of line of the map is: highway, major road, minor road, path, railway, boundary, contour and index contour, each a piece of the line as the map draws it, on the color of the map's land. The map's style says how a line is drawn at each zoom, and a sheet is drawn at one. The browser, which has the style, says what each line's color, width, dashes and edges are at that zoom; the API draws them. A line is in the legend when its layer is exported and draws something at that zoom. What the browser sends is made into lines before it is kept: what is not a line is left out, and the map is exported. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * A legend has no line too thin to see, and no name longer than its room A style brings a line in from nothing as the map is zoomed in: a path is a few hundredths of a point wide on a quadrangle near the equator. It was in the legend, drawn several times wider than on the map. A line under a tenth of a point is now left out, by the browser and by the API. What a line is called is cut to the room beside its line, as the name of an adjoining quadrangle is. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * Fixes to the map PDF export: wrong neighbors, leftover exports, exports which outlive the map (#989) * A sheet names what is around it, and its edges, by where they are and not a hair from there The edges the browser gives of a map are a hair outside its quadrangle. The quadrangles around a map whose middle is on the edge two share (every other map of Alaska) were those around the map west of it, and one of them was the map itself. An edge a hair south of the equator was 0 degrees south, a map which ended where the UTM zones do had no grid, and a sheet dated in the last year there is could not be made. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * An export is removed when it is over, and its work stops when the API does The images and the PDF of an export were kept until the next export began, however long that was. The browser now has the export removed however it ends, and old exports are removed when a PDF is made too. The work of a Job which ended because the API was stopping went on to the end of the PDF, with the API waiting for it. What is said of an export, and each image, is written whole. Text which is no text is taken out when an export begins, not found by the Job. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * An export stops with its map, and waits for a page which can not be seen Leaving the map while it was exported left the export going, to fail or to hand a PDF to another page; so did closing the picker, which is now left open. A PDF which arrives after the export was stopped is not saved. The time a page is behind another tab is not counted against a map which is drawing. The name of a road is one line. A photograph which ends a little inside the map is not made a JPEG with a black strip. An accent in the name of a quadrangle is not a gap in the name of its file. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * A PDF is not kept from who made it by a server slow to remove the export Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * An exported map has its water over its parks (#990) * A sheet names what is around it, and its edges, by where they are and not a hair from there The edges the browser gives of a map are a hair outside its quadrangle. The quadrangles around a map whose middle is on the edge two share (every other map of Alaska) were those around the map west of it, and one of them was the map itself. An edge a hair south of the equator was 0 degrees south, a map which ended where the UTM zones do had no grid, and a sheet dated in the last year there is could not be made. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * An export is removed when it is over, and its work stops when the API does The images and the PDF of an export were kept until the next export began, however long that was. The browser now has the export removed however it ends, and old exports are removed when a PDF is made too. The work of a Job which ended because the API was stopping went on to the end of the PDF, with the API waiting for it. What is said of an export, and each image, is written whole. Text which is no text is taken out when an export begins, not found by the Job. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * An export stops with its map, and waits for a page which can not be seen Leaving the map while it was exported left the export going, to fail or to hand a PDF to another page; so did closing the picker, which is now left open. A PDF which arrives after the export was stopped is not saved. The time a page is behind another tab is not counted against a map which is drawing. The name of a road is one line. A photograph which ends a little inside the map is not made a JPEG with a black strip. An accent in the name of a quadrangle is not a gap in the name of its file. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * An exported map has its water over its parks A layer of the PDF was one image, drawn where the topmost of its layers is. The parks of Land Cover are under the water on the map and its piers are over it, so all of Land Cover was drawn over the water: a lake in a park, or among houses, was not on the PDF. A second basemap file put its water over the first's buildings. A layer is now as many images as it must be, each where it is on the map, in one layer of the PDF: a part which fills an area stays where it is, and a part which is only lines is drawn with the next. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * A PDF is not kept from who made it by a server slow to remove the export Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * A map file which is of somewhere else is not drawn when a map is exported Each basemap file is several images of an exported map, and most files of a library are of other places: each was an image of nothing, counted against how many images an export may be and how sharp it may be. A file whose own edges are outside the quadrangle is now left out before the layers are counted. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * An exported map has its streams (#991) A basemap draws streams from zoom 14, and service roads and taxiways from 13. A quadrangle is drawn a little short of that (zoom 13.8 at 40 degrees, 13.2 at 61), from tiles which have them: none were on the PDF. What is drawn from a zoom no deeper than a quadrangle's detail is drawn. Names are left as they are. Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * An export which is cancelled stops at once, in the middle of a request (#993) * An export which is cancelled stops at once, in the middle of a request An export was stopped between two requests. The image of a layer is tens of megabytes, which is a long request on a slow connection, and a request to a server which has stopped answering never ends: the export said it was stopping until then. Cancel, and leaving the map, now end the request. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * The requests an export makes after its layers are sent are seen to be stopped too Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * A map is exported as it is printed: on white, in dark lines (#996) * A map is exported as it is printed: on white, in dark lines The light and dark maps are made for a screen. On paper their roads are white on pale gray and their streams half a pixel of pale blue. The printed map is a third look, of the map's own: white ground, dark road edges and names, blue streams wide enough to see, paths drawn, brown contours. It is what a map is exported in unless Light or Dark is chosen where the quadrangle is, and the choice is remembered in the browser. While a quadrangle is chosen the map is shown in the look it will be exported in. The light and dark maps are as they were. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * Changing the look of the map does not move it What is done when a new style is in place was asked for after the style was set. A style which only differs from the one the map has is in place before that, so it was done at the next change of style instead, with where the map had been at this one: the map went back there when a quadrangle was chosen, and again when it was not. It is asked for first, and the map is not moved at all. An export no longer waits for the map to be still, which it need not and which nothing ended. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * The colors of the printed map are chosen where its layers are (#997) * The colors of the printed map are chosen where its layers are While a quadrangle is chosen and the map is shown as it is printed, each layer of the layer control has its color beside it, to be changed: one for most, three for the land (what grows, what was built, sand and ice) and two for names (the name, and what is around it). A basemap has several colors where one is chosen, which are shades of it. The edge of a road stays dark. What is chosen is remembered in the browser, and can be put back. The map on the screen is as it was. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * A ramp on a bridge is the color of a major road, as it is off one A basemap draws a ramp on a bridge in the color of a minor road on a bridge: on the printed map it was white where the rest of the ramp is yellow, and followed the color chosen for minor roads. What puts the colors back is a button, which a keyboard reaches. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * A layer with nothing in it does not make an exported map less sharp (#1001) * A layer with nothing in it does not make an exported map less sharp How sharp a map may be depends on how many images it is, and every group of layers was counted: a basemap file with no runway, highway or boundary in the quadrangle was images of nothing, drawn, sent, and counted against the rest. A map of 23 images asked for at 400 DPI was made at 330. The groups are now looked at first, on a small map, and only those with something in them are drawn, as sharp as that many may be. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * An image which arrives after its export was removed is not kept, and one mil is a mil Stopping an export while an image was on its way left the image behind it, in a directory nothing would remove for an hour. An angle of one mil was "1 mils" on the sheet. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * The map which is exported from is made as sharp as it is to be (#1002) * The map which is exported from is made as sharp as it is to be The groups of a map are looked at on a small map, which was then made larger to draw them. A browser lets a canvas be larger when it is made than it lets one be made larger afterwards: a quadrangle asked for at 400 DPI was 33 million pixels, 288 DPI at 40 degrees, less than before the groups were looked at first. The small map is now removed, and the map which is exported from is made in its place. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * A canvas the browser stopped drawing on is not a layer with nothing in it A canvas whose drawing was taken away by the browser reads as clear, and a group looked at then was left out of the PDF without a word: the export fails instead. An export stopped as the last group is looked at does not make the map it would have been drawn on. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * An exported map says where it is in a projection Adobe Acrobat knows (#1003) * An exported map says where it is in a projection Adobe Acrobat knows The page was said to be in Web Mercator, which it is drawn in. Acrobat does not know Web Mercator, in the words of the standard or in Esri's: it took the page for a map, and put every place on it at 0, 0. The page is now said to be Mercator on the WGS 84 ellipsoid, which it reads, with the same corners. Between the corners the two differ by under three centimeters on the ground. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * How well a page is said to be where it is, is measured from what the PDF says The test of it measured two formulas of its own. It now reads the corners and the projection from the PDF and does a reader's sum with them. The few centimeters are of a quadrangle: a taller page is less well said, by the square of how tall it is, which the comment now says and a test measures. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> * The help says how a map is exported as a PDF (#1004) The help of the map is five pages, one of them the PDF export. Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-20 17:22:21 -07:00
"""Making the PDF of an exported map as a Job.
A PDF of a map is a minute or two of work on a Raspberry Pi: too long for a request to wait on. So the browser
describes the map and is given the id of an export; it sends the image of each layer, one a request (together they
are a hundred megabytes, more than one request may be); then it asks for the PDF to be made, and is given the id of
a Job (see `wrolpi.jobs`). It watches the Job, and when the Job is done it fetches the PDF.
An export is a directory, which holds what was said of the map, the images until the PDF is made of them, and then
the PDF. It is on a disk, not in memory. It is removed when the person exporting stops it, when its Job is
cancelled, or when it is old; never while its Job is waiting or running.
"""
import asyncio
import json
import os
import pathlib
import re
import shutil
import stat
import threading
import time
import uuid
from typing import List, Optional
from modules.map import export
from wrolpi.common import logger
from wrolpi.errors import UnknownJob
from wrolpi.jobs import (CANCELLED, COMPLETE, PENDING, RUNNING, cancel_job, get_current_job, get_job,
register_job)
logger = logger.getChild(__name__)
__all__ = ['ExportNotFound', 'ExportNotReady', 'ExportStorageError', 'create_export', 'save_layer', 'start_export',
'get_export_pdf', 'remove_export', 'remove_old_exports', 'map_export_pdf_job', 'EXPORT_ROOT', 'EXPORT_KEEP_SECONDS', 'MAX_LAYER_BYTES']
# Where exports are kept. Not /tmp, which is memory on a Raspberry Pi. /var/tmp is a directory anyone on the
# machine may make things in, so this one is only used when it is a real directory of this user's (see `_root`).
EXPORT_ROOT = pathlib.Path('/var/tmp/wrolpi-map-export')
# An export which is this old is removed: its PDF was fetched long ago, or never will be.
EXPORT_KEEP_SECONDS = 60 * 60
# The most one image of a layer may be. A layer of a quadrangle at 400 DPI is 20 to 30 MB as a PNG.
MAX_LAYER_BYTES = 64 * 1024 * 1024
JOB_NAME = 'map_export_pdf'
META_FILE = 'meta.json'
PDF_FILE = 'map.pdf'
EXPORT_ID = re.compile(r'^[0-9a-f]{32}$')
PNG_MAGIC, JPEG_MAGIC = b'\x89PNG\r\n\x1a\n', b'\xff\xd8\xff'
class ExportNotFound(Exception):
"""There is no such export: it was never made, or it was old and has been removed."""
class ExportNotReady(Exception):
"""The export is not as far along as it was asked to be."""
class ExportStorageError(Exception):
"""Where exports are kept can not be used."""
class ExportCancelled(Exception):
"""The Job which was making the PDF was cancelled."""
def _is_own_directory(path: pathlib.Path) -> bool:
"""Whether a path is a directory itself (not a link to one) which this user owns."""
try:
status = os.lstat(path)
except OSError:
return False
return stat.S_ISDIR(status.st_mode) and status.st_uid == os.getuid()
def _root(create: bool = False) -> Optional[pathlib.Path]:
"""Where exports are kept, or None when there is no such directory (and it is not to be made).
It is made for this user alone: the images of a map are whatever the user has on their map.
@raise ExportStorageError: When what is there is not a directory of this user's: a link there, which another
user of the machine may have made, would have exports written (and old ones removed) wherever it leads."""
if create:
try:
os.mkdir(EXPORT_ROOT, 0o700)
except FileExistsError:
pass
except OSError as e:
raise ExportStorageError(f'The directory of map exports can not be made: {EXPORT_ROOT}') from e
if not os.path.lexists(EXPORT_ROOT):
return None
if not _is_own_directory(EXPORT_ROOT):
raise ExportStorageError(f'The directory of map exports is not a directory of this user: {EXPORT_ROOT}')
if create and stat.S_IMODE(os.lstat(EXPORT_ROOT).st_mode) != 0o700:
# One which was made before it was made for this user alone.
os.chmod(EXPORT_ROOT, 0o700)
return EXPORT_ROOT
def _directory(export_id: str) -> pathlib.Path:
"""Where an export is. The id is what `create_export` gave, and nothing else: it becomes part of a path."""
if not EXPORT_ID.match(str(export_id)):
raise ExportNotFound(f'There is no export {export_id!r}')
root = _root()
directory = root / export_id if root else None
if directory is None or not _is_own_directory(directory) or not (directory / META_FILE).is_file():
raise ExportNotFound(f'There is no export {export_id!r}')
return directory
def _read_meta(directory: pathlib.Path) -> dict:
return json.loads((directory / META_FILE).read_text())
def _write_whole(path: pathlib.Path, contents: bytes):
"""Write a file so that whoever reads it reads all of it, or what was there before: never a part."""
partial = path.with_name(f'{path.name}.{uuid.uuid4().hex}.part')
try:
partial.write_bytes(contents)
os.replace(partial, path)
finally:
partial.unlink(missing_ok=True)
def _write_meta(directory: pathlib.Path, meta: dict):
_write_whole(directory / META_FILE, json.dumps(meta).encode())
def _layer_file(directory: pathlib.Path, index: int) -> pathlib.Path:
return directory / f'layer-{index:03d}'
def _job_status(meta: dict) -> Optional[str]:
"""The status of the Job of an export, or None when it has none: it was not started, or the API has
restarted since and has forgotten the Job."""
if not meta.get('job_id'):
return None
try:
return get_job(meta['job_id'])['status']
except UnknownJob:
return None
def remove_old_exports(now: float = None):
"""Remove the exports which are done with. An export is a hundred megabytes until its PDF is made, and tens
after.
An export whose Job is waiting or running is kept however old it is: its images are what the Job is to work
from, and it may wait behind another Job for longer than an export is kept. One whose Job was cancelled is
removed at once. Any other is removed when it is old. Only what is an export is touched."""
now = time.time() if now is None else now
root = _root()
if root is None:
return
for directory in root.iterdir():
try:
if not EXPORT_ID.match(directory.name) or not _is_own_directory(directory):
continue
try:
status = _job_status(_read_meta(directory))
except (OSError, ValueError):
# No description which can be read: not an export any Job can use.
status = None
if status in (PENDING, RUNNING):
continue
if status == CANCELLED or now - directory.stat().st_mtime > EXPORT_KEEP_SECONDS:
shutil.rmtree(directory, ignore_errors=True)
except OSError as e:
logger.warning(f'Could not remove the old map export {directory}', exc_info=e)
def remove_export(export_id: str):
"""Stop an export, and remove it: the person exporting does not want it.
Its Job is cancelled when it is waiting or running. A running Job stops between two layers and removes the
images itself; until then they are in use, so what is left of the export is removed later, with the old ones."""
directory = _directory(export_id)
meta = _read_meta(directory)
status = _job_status(meta)
if status in (PENDING, RUNNING):
cancel_job(meta['job_id'])
if status != RUNNING:
shutil.rmtree(directory, ignore_errors=True)
def create_export(layers: List[str], width: int, height: int, dpi: float, title: Optional[str] = None,
bounds=None, sheet: Optional[export.Sheet] = None) -> str:
"""Begin an export of a map of these layers (their names, from the bottom up). Returns its id.
@raise InvalidExport: When such a map can not be exported; no image has been sent yet."""
bounds = export.check_export(len(layers), width, height, dpi, bounds, sheet)
root = _root(create=True)
remove_old_exports()
export_id = uuid.uuid4().hex
directory = root / export_id
os.mkdir(directory, 0o700)
_write_meta(directory, dict(
layers=[str(i) for i in layers], width=width, height=height, dpi=dpi, title=title, bounds=bounds,
sheet=dict(name=sheet.name, credits=list(sheet.credits), date=sheet.date,
contour_interval=sheet.contour_interval, contour_unit=sheet.contour_unit,
legend=sheet.legend) if sheet else None,
job_id=None,
))
return export_id
def save_layer(export_id: str, index: int, image: bytes):
"""Keep the image of one layer of an export.
@raise InvalidExport: When it is not an image of a layer of that export."""
directory = _directory(export_id)
meta = _read_meta(directory)
if meta['job_id']:
raise export.InvalidExport('The PDF of this map is already being made')
if not 0 <= index < len(meta['layers']):
raise export.InvalidExport(f'The map has {len(meta["layers"])} layers; there is no layer {index}')
if not image or len(image) > MAX_LAYER_BYTES:
raise export.InvalidExport(f'The image of the layer {meta["layers"][index]!r} is empty, or is too large')
if not image.startswith((PNG_MAGIC, JPEG_MAGIC)):
raise export.InvalidExport(f'The layer {meta["layers"][index]!r} is not a PNG or JPEG image')
_write_whole(_layer_file(directory, index), image)
if not (directory / META_FILE).is_file():
# The export was removed while this image was on its way: the image is not left behind it.
shutil.rmtree(directory, ignore_errors=True)
raise ExportNotFound(f'There is no export {export_id!r}')
def start_export(export_id: str) -> str:
"""Queue the Job which makes the PDF of an export, all of whose images have been sent. Returns the Job's id;
asked again, the same Job's."""
directory = _directory(export_id)
meta = _read_meta(directory)
if meta['job_id']:
return meta['job_id']
missing = [name for index, name in enumerate(meta['layers']) if not _layer_file(directory, index).is_file()]
if missing:
raise ExportNotReady(f'The images of {len(missing)} layers have not been sent: {", ".join(missing[:5])}')
job_id = map_export_pdf_job.enqueue(description=f'Map PDF: {meta["title"] or "a map"}', export_id=export_id)
meta['job_id'] = job_id
_write_meta(directory, meta)
return job_id
def get_export_pdf(export_id: str) -> pathlib.Path:
"""The PDF of an export.
@raise ExportNotReady: When it has not been made (yet, or because the Job failed or was cancelled)."""
directory = _directory(export_id)
pdf = directory / PDF_FILE
if not pdf.is_file():
job_id = _read_meta(directory).get('job_id')
status = get_job(job_id)['status'] if job_id else 'not started'
raise ExportNotReady(f'The PDF has not been made (the job is {status})' if status != COMPLETE
else 'The PDF is gone')
return pdf
def _make_pdf(directory: pathlib.Path, job, stop: threading.Event) -> dict:
"""Make the PDF of an export from its images, which are removed whether or not it could be made.
The work stops between two layers when the Job is cancelled, and when `stop` is set: the Job was ended by
something other than its being cancelled (the API is stopping), which the Job itself does not say."""
meta = _read_meta(directory)
layers = [export.ExportLayer(name=name, path=_layer_file(directory, index))
for index, name in enumerate(meta['layers'])]
sheet = export.Sheet(**meta['sheet']) if meta['sheet'] else None
def stopped() -> bool:
return stop.is_set() or job.cancel_requested()
def progress(done: int, steps: int):
# Between two layers is where the work can be stopped.
if stopped():
raise ExportCancelled()
job.set_progress(100 * done / steps)
try:
pdf = export.build_layered_pdf(layers, meta['width'], meta['height'], meta['dpi'], meta['title'],
meta['bounds'], sheet, progress=progress)
# Under another name until all of it is written: a PDF which is there is a whole one.
partial = directory / f'{PDF_FILE}.part'
partial.write_bytes(pdf)
if stopped():
partial.unlink(missing_ok=True)
raise ExportCancelled()
partial.rename(directory / PDF_FILE)
return dict(export_id=directory.name, size=len(pdf))
finally:
for layer in layers:
layer.path.unlink(missing_ok=True)
@register_job(JOB_NAME)
async def map_export_pdf_job(export_id: str):
"""Make the PDF of an export. The work is in a thread: this process has other things to keep doing.
A Job which is cancelled is not over until that thread has stopped: Jobs are run one at a time so that two of
them do not work a Raspberry Pi at once, and the next is not to start beside the work of this one."""
directory = _directory(export_id)
job = get_current_job()
job.log(f'Making the PDF of the map export {export_id}')
# Old exports are removed here too: an export may be the last for a long while.
await asyncio.to_thread(remove_old_exports)
stop = threading.Event()
work = asyncio.ensure_future(asyncio.to_thread(_make_pdf, directory, job, stop))
try:
# Shielded: cancelling this Job must not let go of the thread, which can not be stopped from here.
result = await asyncio.shield(work)
except asyncio.CancelledError:
# The thread is told to stop, which it does between two layers. Wait for it to: an API which is stopping
# waits for its threads, and is not to wait for all of a PDF.
stop.set()
await asyncio.wait({work})
if not work.cancelled():
work.exception()
raise
except ExportCancelled:
# The work saw that the Job was cancelled before the Job's runner did. The runner is who ends a cancelled
# Job (it looks every second), so wait for it to.
await asyncio.Event().wait()
raise
job.log(f'The PDF is {result["size"]} bytes')
return result