Merge pull request #533 from SuffolkLITLab/copilot/setup-copilot-instructions

Add comprehensive GitHub Copilot instructions for documentation build process

Author: nonprofittechy

.github/copilot-instructions.md

@@ -0,0 +1,124 @@
+# Document Assembly Line Documentation
+
+The Document Assembly Line Documentation is a Docusaurus-based documentation website that combines automatic Python API documentation generation with manually authored guides for the Document Assembly Line project by Suffolk LIT Lab.
+
+Always reference these instructions first and fallback to search or bash commands only when you encounter unexpected information that does not match the info here.
+
+## Working Effectively
+
+Bootstrap, build, and test the repository:
+- Install Python dependencies: `pip install docspec_python==2.2.1 git+https://github.com/nonprofittechy/pydoc-markdown@escape-brackets` -- takes 20 seconds
+- Clone external documentation sources (required for full build):
+  ```bash
+  cd .. # go up one directory from repo root
+  git clone https://github.com/SuffolkLITLab/docassemble-AssemblyLine.git
+  git clone https://github.com/SuffolkLITLab/FormFyxer.git  
+  git clone https://github.com/SuffolkLITLab/docassemble-ALToolbox.git
+  git clone https://github.com/SuffolkLITLab/docassemble-EFSPIntegration.git
+  cd docassemble-AssemblyLine-documentation # return to repo root
+  ```
+- Generate Python API documentation: `pydoc-markdown` -- takes 2 seconds
+- Fix FormFyxer case sensitivity: `rm -rf docs/components/formfyxer` (if the directory exists)
+- Install Node.js dependencies: `PUPPETEER_SKIP_DOWNLOAD=true npm install` -- takes 20 seconds
+- Build the documentation: `npm run build` -- **WILL FAIL with PDF generation error** - this is expected behavior in CI environments. The HTML build succeeds and creates usable documentation in the `build/` directory.
+
+Run the documentation website:
+- ALWAYS run the bootstrapping steps first.
+- Development server: `npm run start` -- starts on http://localhost:3000, takes 30 seconds to be ready
+- Production server: `npm run serve` -- serves built files on http://localhost:3000
+
+## Critical Build Information
+
+- **NEVER CANCEL** the `npm run build` command - it takes 60 seconds and may appear to hang during webpack compilation
+- **ALWAYS** use `PUPPETEER_SKIP_DOWNLOAD=true npm install` due to network restrictions preventing Chrome download
+- **PDF generation ALWAYS FAILS** in CI environments due to Puppeteer Chrome download restrictions - this is expected behavior
+- FormFyxer case sensitivity: pydoc-markdown creates duplicate `FormFyxer` and `formfyxer` directories, causing build warnings. If the build fails with case sensitivity errors, run: `rm -rf docs/components/formfyxer`
+- The build generates warnings about broken anchors - these are expected and do not prevent successful builds
+- Complete build workflow: `pydoc-markdown && rm -rf docs/components/formfyxer && npm run build`
+
+## Validation
+
+- ALWAYS manually validate any new code by starting the development server and navigating to affected pages
+- Test both development mode (`npm run start`) and production build (`npm run build && npm run serve`)
+- The site should load at http://localhost:3000 with a blue-themed homepage featuring "Open-source tools for court forms, guided interviews, and e-filing"
+- Navigation should work between homepage, Get started, and Documentation sections
+- Always run `npm run clear` before building if you encounter unexpected webpack errors
+
+## Common Tasks
+
+The following are outputs from frequently run commands. Reference them instead of viewing, searching, or running bash commands to save time.
+
+### Repository root structure
+```
+.
+├── .git/
+├── .github/
+│   └── workflows/
+│       ├── deploy.yml
+│       └── test-deploy.yml
+├── .gitignore
+├── README.md
+├── babel.config.js
+├── docs/
+├── docusaurus.config.js
+├── package-lock.json
+├── package.json
+├── pydoc-markdown.yml
+├── sidebars.js
+├── src/
+└── static/
+```
+
+### Key configuration files
+- `docusaurus.config.js`: Main Docusaurus configuration including plugins, themes, and site metadata
+- `pydoc-markdown.yml`: Configuration for extracting Python API documentation from external repositories
+- `sidebars.js`: Navigation structure for documentation sections
+- `package.json`: Node.js dependencies and scripts
+
+### Build process sequence
+1. Python dependencies installation (~20 seconds)
+2. External repositories cloning (~5 seconds)
+3. Python API documentation generation via pydoc-markdown (~2 seconds)
+4. Node.js dependencies installation (~20 seconds)
+5. Docusaurus build compilation (~60 seconds)
+
+### Known working commands
+- `npm run clear` -- clears build cache, takes 1 second
+- `npm run start` -- development server, ready in 30 seconds
+- `npm run build` -- production build, takes 60 seconds
+- `npm run serve` -- serves built files
+- `npm run swizzle` -- customize Docusaurus components
+- `npm run deploy` -- deploys to GitHub Pages (requires GIT_USER and USE_SSH)
+
+### Development workflow
+1. Make changes to markdown files in `docs/` directory
+2. Test with `npm run start` for live reload during development
+3. Build with `npm run build` to test production build
+4. Check for build warnings and broken links
+5. Test navigation and content rendering with `npm run serve`
+
+### External dependencies
+The site pulls API documentation from these repositories:
+- `docassemble-AssemblyLine`: Main Assembly Line framework
+- `FormFyxer`: PDF and DOCX manipulation tools
+- `docassemble-ALToolbox`: Additional utility functions
+- `docassemble-EFSPIntegration`: E-filing integration components
+
+### Troubleshooting
+- If npm install fails: use `PUPPETEER_SKIP_DOWNLOAD=true npm install`
+- If build fails with FormFyxer case errors: `rm -rf docs/components/formfyxer`
+- If webpack compilation appears stuck: wait at least 60 seconds before investigating
+- If development server fails to start: run `npm run clear` first
+- **PDF generation always fails in CI** due to Chrome/Puppeteer restrictions - this is normal and expected
+- For local PDF generation: ensure Chrome is installed and accessible to Puppeteer, or disable `autoBuildPdfs` in `docusaurus.config.js`
+
+### CI/CD Pipeline
+GitHub Actions automatically:
+1. Installs Python 3.11 and required packages
+2. Clones external documentation repositories
+3. Runs pydoc-markdown for API documentation
+4. Installs Node.js 20 and npm dependencies
+5. Builds the site with `npm run build`
+6. Deploys to GitHub Pages on main branch pushes
+
+Test deployment runs the same process for pull requests without the final deploy step.

docs/components/ALToolbox/llms.md

@@ -7,21 +7,22 @@ title: ALToolbox.llms
 
 A light wrapper on the OpenAI chat endpoint.
 
-Includes support for token limits, error handling, and moderation queue.
-
-It is also possible to specify an alternative model, and we support GPT-4-turbo's JSON
-mode.
-
-As of today (1/2/2024) JSON mode requires the model to be set to "gpt-4-1106-preview" or "gpt-3.5-turbo-1106"
+Includes support for token limits, minimal error handling, and moderation.
 
 **Arguments**:
 
 - `system_message` _str_ - The role the chat engine should play
 - `user_message` _str_ - The message (data) from the user
 - `openai_client` _Optional[OpenAI]_ - An OpenAI client object, optional. If omitted, will fall back to creating a new OpenAI client with the API key provided as an environment variable
 - `openai_api` _Optional[str]_ - the API key for an OpenAI client, optional. If provided, a new OpenAI client will be created.
-- `temperature` _float_ - The temperature to use for the GPT-4-turbo API
-- `json_mode` _bool_ - Whether to use JSON mode for the GPT-4-turbo API
+- `temperature` _float_ - The temperature to use for the GPT API
+- `json_mode` _bool_ - Whether to use JSON mode for the GPT API. Requires the word `json` in the system message, but will add if you omit it.
+- `model` _str_ - The model to use for the GPT API
+- `messages` _Optional[List[Dict[str, str]]]_ - A list of messages to send to the chat engine. If provided, system_message and user_message will be ignored.
+- `skip_moderation` _bool_ - Whether to skip the OpenAI moderation step, which may save seconds but risks banning your account. Only enable when you have full control over the inputs.
+- `openai_base_url` _Optional[str]_ - The base URL for the OpenAI API. Defaults to value provided in the configuration or "https://api.openai.com/v1/".
+- `max_output_tokens` _Optional[int]_ - The maximum number of tokens to return from the API. Defaults to 16380.
+- `max_input_tokens` _Optional[int]_ - The maximum number of tokens to send to the API. Defaults to 128000.
 
 
 **Returns**:
@@ -44,7 +45,7 @@ Extracts fields from text.
 
 #### match\_goals\_from\_text
 
-Read's a user's message and determines whether it meets a set of goals, with the help of an LLM.
+Reads a user's message and determines whether it meets a set of goals, with the help of an LLM.
 
 **Arguments**:
 
@@ -232,3 +233,59 @@ Returns the next unsatisfied goal, along with a follow-up question to ask the us
 
 Returns a draft response that synthesizes the user's responses to the questions.
 
+#### provide\_feedback
+
+Returns feedback to the user based on the goals they satisfied.
+
+## IntakeQuestion Objects
+
+```python
+class IntakeQuestion(DAObject)
+```
+
+A class to represent a question in an LLM-assisted intake questionnaire.
+
+**Attributes**:
+
+- `question` _str_ - The question to ask the user
+- `response` _str_ - The user's response to the question
+
+## IntakeQuestionList Objects
+
+```python
+class IntakeQuestionList(DAList)
+```
+
+Class to help create an LLM-assisted intake questionnaire.
+
+The LLM will be provided a free-form set of in/out criteria (like that
+provided to a phone intake worker), an initial draft question from the user,
+and then guide the user through a series of follow-up questions to gather only
+enough information to determine if the user meets the criteria.
+
+In/out criteria are often pretty short, so we do not make or support
+embeddings at the moment.
+
+**Attributes**:
+
+- `criteria` _Dict[str, str]_ - A dictionary of criteria to match, indexed by problem type
+- `problem_type_descriptions` _Dict[str, str]_ - A dictionary of descriptions of the problem types
+- `problem_type` _str_ - The type of problem to match. E.g., a unit/department inside the law firm
+- `initial_problem_description` _str_ - The initial description of the problem from the user
+- `initial_question` _str_ - The original question posed in the interview
+- `question_limit` _int_ - The maximum number of follow-up questions to ask the user. Defaults to 10.
+- `model` _str_ - The model to use for the GPT API. Defaults to gpt-4.1.
+- `max_output_tokens` _int_ - The maximum number of tokens to return from the API. Defaults to 4096
+- `llm_role` _str_ - The role the LLM should play. Allows you to customize the script the LLM uses to guide the user.
+  We have provided a default script that should work for most intake questionnaires.
+- `llm_user_qualifies_prompt` _str_ - The prompt to use to determine if the user qualifies. We have provided a default prompt.
+- `out_of_questions` _bool_ - Whether the user has run out of questions to answer
+- `qualifies` _bool_ - Whether the user qualifies based on the criteria
+
+#### need\_more\_questions
+
+Returns True if the user needs to answer more questions, False otherwise.
+
+Also has the side effect of checking the user's most recent response to see if it satisfies the criteria
+and updating both the next question to be asked and the current qualification status.
+

docs/components/ALToolbox/misc.md

@@ -28,7 +28,7 @@ you more control over the icon that is inserted.
 
 - `icon` - a string representing a fontawesome icon. The icon needs to be in the
   [free library](https://fontawesome.com/search?o=r&m=free).
-- `color` - can be any [Bootstrap color variable](https://getbootstrap.com/docs/5.0/utilities/colors/).
+- `color` - can be any [Bootstrap color variable](https://getbootstrapc.mo/docs/4.0/utilities/colors).
   For example: `primary`, `secondary`, `warning`
 - `color_css` - allows you to use a CSS code to represent the color, e.g., `blue`, or ``fff`` for black
 - `size` - used to control the [fontawesome size](https://fontawesome.com/v6.0/docs/web/style/size)
@@ -148,3 +148,114 @@ of privileges.
 
   HTML for a grid of buttons
 
+#### none\_to\_empty
+
+If the value is None or "None", return a DAEmpty value. Otherwise return the value.
+
+This is useful for filling in a template and to prevent the word None from appearing in the output. For example,
+when handling a radio button that is not required and left unanswered.
+
+A DAEmpty value appears as an empty string in the output. You can also safely transform it or use any method on it
+without raising an error.
+
+**Arguments**:
+
+- `val` - the value to check
+
+**Returns**:
+
+  a DAEmpty if the value is None, otherwise the value
+
+#### option\_or\_other
+
+If the variable is set to 'Other', return the value of the 'other' variable. Otherwise return the value of the variable.
+
+This is useful for filling in a template and to prevent the word 'Other' from appearing in the output.
+
+**Arguments**:
+
+- `variable_name` - the name of the variable to check
+- `other_variable_name` - the name of the variable to return if the value of the first variable is 'Other'
+
+**Returns**:
+
+  the value of the variable if it is not 'Other', otherwise the value of the other variable
+
+#### true\_values\_with\_other
+
+Return a list of values that are True, with the value of the 'other' variable appended to the end of the list.
+
+This is useful for filling in a template and to prevent the word 'Other' from appearing in the output.
+
+**Arguments**:
+
+- `variable` - the dictionary of variables to check
+- `other_variable_name` - the name of the variable (as a string) to return if the value of the first variable is 'Other'
+
+**Returns**:
+
+  a list of values that are True, with the value of the 'other' variable appended to the end of the list.
+
+#### include\_a\_year
+
+Validates whether the input text contains at least one 4-digit sequence
+that occurs within a range of ~ 200 years, indicating a valid "year"
+for an event that should be reported on most court forms, like a birthdate
+or a moving date.
+
+Returns True if found, otherwise raises a DAValidationError.
+
+#### is\_leap\_year
+
+Helper function for `age_in_years` to determine if a year is a leap year.
+
+**Arguments**:
+
+- `year` - The year to check.
+
+**Returns**:
+
+  True if the year is a leap year, False otherwise.
+
+#### age\_in\_years
+
+Calculate the age in years from a date (treated like a date of birth).
+
+**Arguments**:
+
+- `the_date` - A string or DADateTime object representing the date of birth.
+
+**Returns**:
+
+  The age in years as an integer.
+
+#### format\_date\_if\_defined
+
+Format a date string if it is defined, otherwise return an empty string.
+
+Passes all additional arguments to the `format_date` function.
+
+**Arguments**:
+
+- `date_object_name` - The date string to format.
+- `*pargs` - Additional positional arguments to pass to `format_date`.
+- `default` - A default string to return if `date_object_name` is not defined.
+- `**kwargs` - Additional keyword arguments to pass to `format_date`. E.g., format="yyyy-MM-dd"
+  
+
+**Returns**:
+
+  A formatted date string if `date_object_name` is defined, otherwise an empty string.
+  
+
+**Example**:
+
+  
+  >>> format_date_if_defined("users[0].birthdate", format='yyyy-MM-dd')
+  
+  Returns a formatted date string if "users[0].birthdate" is defined, otherwise returns an empty string.
+  
+  >>> format_date_if_defined("users[0].birthdate", default="No date provided", format='yyyy-MM-dd ')
+  
+  Returns a formatted date string followed by one space if "users[0].birthdate" is defined, otherwise returns "No date provided". (Note space is added to the format="..." parameter)
+