formulae-capitains-nemo documentation

formulae-capitains-nemo

This is the class extension “NemoFormulae” for flask_nemo. A working instance of this extension for the Formulae - Litterae - Chartae Project can be found at https://werkstatt.formulae.uni-hamburg.de.

Getting Started

Further information:

• https://github.com/capitains/tutorial-nemo

• The app is configured via formulae/app.py

• Complete documentation should be done using Sphinx

Running the app locally :computer::

1. Preliminary setup steps:

Only need to be executed the before the first time running the app:

1. Clone the repositories:

   1. git clone formulae-capitains-nemo (code-base)

   2. git clone formulae-corpora (texts) (ideally in the same folder e.g., git as the code base)

2. Create a Python virtualenv (e.g., virtualenv --python=python3 .venv)

3. Only if needed: Set the environment variable CORPUS_FOLDERS and re-start the app.

2. Start the app:

1. activate the virtualenv (e.g., source .venv/bin/activate)

2. install the requirements via pip install -r requirements.txt within in the venv and from the formulae-capitains-nemo folder

3. Optional: set-up Elastic Search via: .env

4. For local development: ELASTICSEARCH_URL = "http://localhost:9200" -> requires: local elastic search instance

5. If the requirements have been installed properly, you can launch python3 app.py within the venv and in formulae-capitains-nemo folder

6. Reach the site via 127.0.0.1:5000

Run Elastic Search local

1. Make sure that you have a few Gigabytes of RAM free

2. cd formulae-capitains-nemo folder

3. docker-compose up

4. es8 exited with code 137 -> Not enough memory free

Running with Docker Compose :whale2:

The application can be started locally using Docker Compose. The setup includes:

• Elasticsearch – search index backend

• Redis – temporary storage for search workflows

• formulae_corpora – helper container that clones/updates the XML corpus and can rebuild the search index

• nemo – the Flask web application

Requirements

• Docker

• Docker Compose

Environment variables

Create a .env file in the project root:

  ELASTICSEARCH_URL=http://elasticsearch:9200
  FORMULAE_CORPORA_REPO_URL=<repository>
  GITHUB_TOKEN=<github-token>
  FORMULAE_CORPORA_REF=
  REBUILD_ELASTICSEARCH=false

Startup sequence

  docker compose up -d elasticsearch redis
  docker compose run --rm formulae_corpora
  docker compose up -d nemo

:computer: Application URL: http://localhost:5000

Stop

  docker compose down

How are static files handled?

1. https://flask.palletsprojects.com/en/2.3.x/quickstart/#static-files

2. I do recommend to add /static and /robots.txt to your nginx configuration, so that are served directly without passing through the application.

How to run the SPHINX documentation locally:

1. Install sphinx: https://www.sphinx-doc.org/en/master/usage/installation.html

   • For Debin/Ubuntu the OS-specific package manager worked best

2. activate the virtualenv (e.g., source .venv/bin/activate)

3. install the requirements via pip install -r requirements_sphinx.txt within in the venv and from the formulae-capitains-nemo folder

4. Build the project: sphinx-build -M html docs/source/ docs/build/ or python -m sphinx -M html docs/source/ docs/build/

5. Open docs/build/html/index.html with your preferred browser: firefox docs/build/html/index.html

Contribution guide

• Currently, we do not follow any specific design pattern. In the future I would to “reduce the weight” of our fat controller formulae/app.py. I have not fully decided on whether I want to have fat models or fat services instead; at the end services vs. models is more a naming thing than a real decision. Alternatively, I could do the MVC-pattern.

• Each new collection should

Run GitHub-actions locally:

1. Install GitHub CLI

2. Install act: gh extension install https://github.com/nektos/gh-act

3. cd git/formulae-capitains-nemo

4. gh act -W '.github/workflows/python-app.yml' or gh act -W '.github/workflows/documentation.yml'

5. Comment out the redis port (gh seems to bring its own redis instance)

Contents:

• Workflow Nemo Design
  • Application Hierarchy
  • Nemo Rendering Workflow
• Traffic management
  • Varnish
    • Restricting parallel resources
    • Cookies
    • Cache Control with max-age
    • Rate limit
  • robots.txt
    • Why add crawl-delay?
    • Why add %2B?
    • Which “good” bots frequently crawled our website?
    • What if a crawler ignores robots.txt
  • nginx
  • flask
• Navigation bar
• formulae nemo
  • NemoFormulae
    • NemoFormulae.CACHED
    • NemoFormulae.after_request()
    • NemoFormulae.call_collate_api()
    • NemoFormulae.check_project_team()
    • NemoFormulae.create_blueprint()
    • NemoFormulae.extract_notes()
    • NemoFormulae.f_insert_in_list()
    • NemoFormulae.f_join_list_values()
    • NemoFormulae.f_random_int()
    • NemoFormulae.f_remove_from_list()
    • NemoFormulae.f_replace_indexed_item()
    • NemoFormulae.get_all_corpora()
    • NemoFormulae.get_first_passage()
    • NemoFormulae.get_locale()
    • NemoFormulae.get_mss_eds()
    • NemoFormulae.get_open_texts()
    • NemoFormulae.get_prev_next_texts()
    • NemoFormulae.get_readable_siblings()
    • NemoFormulae.get_transcriptions()
    • NemoFormulae.highlight_found_sents()
    • NemoFormulae.make_closed_texts()
    • NemoFormulae.make_coins()
    • NemoFormulae.make_collected_colls()
    • NemoFormulae.make_comp_places_list()
    • NemoFormulae.make_dead_url_mapping()
    • NemoFormulae.make_inflected_to_lem_mapping()
    • NemoFormulae.make_lem_to_lem_mapping()
    • NemoFormulae.make_manuscript_notes()
    • NemoFormulae.make_ms_lib_links()
    • NemoFormulae.make_parents()
    • NemoFormulae.ordered_corpora()
    • NemoFormulae.r_accessibility_statement()
    • NemoFormulae.r_add_text_collection()
    • NemoFormulae.r_add_text_collections()
    • NemoFormulae.r_add_text_corpus()
    • NemoFormulae.r_assets()
    • NemoFormulae.r_bibliography()
    • NemoFormulae.r_charter_formulaic()
    • NemoFormulae.r_collection()
    • NemoFormulae.r_contact()
    • NemoFormulae.r_corpus()
    • NemoFormulae.r_corpus_mv()
    • NemoFormulae.r_feedback()
    • NemoFormulae.r_formulae_charter()
    • NemoFormulae.r_formulae_formulae()
    • NemoFormulae.r_groups()
    • NemoFormulae.r_impressum()
    • NemoFormulae.r_lexicon()
    • NemoFormulae.r_man_desc()
    • NemoFormulae.r_man_siglen()
    • NemoFormulae.r_multipassage()
    • NemoFormulae.r_multipassage_authentication_check()
    • NemoFormulae.r_part_groups()
    • NemoFormulae.r_parts()
    • NemoFormulae.r_passage()
    • NemoFormulae.r_pdf()
    • NemoFormulae.r_reading_format()
    • NemoFormulae.r_robots()
    • NemoFormulae.r_set_language()
    • NemoFormulae.r_videos()
    • NemoFormulae.register_font()
    • NemoFormulae.semantic()
    • NemoFormulae.sort_folia()
    • NemoFormulae.sort_katalonien()
    • NemoFormulae.sort_parents()
    • NemoFormulae.sort_transcriptions()
    • NemoFormulae.view_maker()
• Collection
• Deployment
• Docker
• Data structure
• Internationalization
  • Include new internationalized text snippets:
  • How to update existing existing text blocks: