# CommonProjectStructure ## Purpose The common project structure is an attempt for a standardized repository structure for real-world-usecases and -repositories for source-code. It is designed to contain one or more independent (sub)projects which will be called `codeunit`s. So the repository-structure defined by common project structure is applicable for small projects but also for large [Monorepo](https://en.wikipedia.org/wiki/Monorepo)s. This approach follows convention-over-configuration-principles which will result in a very consistent repository-structure. The common project structure is a [opinionated](https://softwareengineering.stackexchange.com/questions/12182) framework for the repository-structure "outside" of the actual program-source-code. The common project structure also recommends things for the actual program-source-code but there are no hard constraints for it. So nevertheless it is possible to use the common project structure with nearly every usual software-project. ### Principles There are some principles underlying this design: 1. Projects must be manageable in a unique consistent way. 2. A project must always be runnable, regardless of whether it is running locally or on a test-system or on a productive-system. The program must not crash, even if a dependent service is not available. For test- and debugging-purposes must be mock-services available. 3. A project must always have a clearly defined version, description, dependency-overview, build-process, linting-ruleset and code-coverage-threshold. 4. A project must always know its version and its supposed environment at build-time. 5. A projects reference must be contained in its repository. 6. Executing the entire build-pipeline including building the project, linting the project, running the testcases, generating the test-coverage and the rest-coverage-report, check the code-coverage-threshold, generating all artifacts and doing all other project-specific-tasks except signing must be doable on every developer-computer. 7. Running the build-pipeline must be idempotent. That means in particular that after running the build-pipeline a second time without changing any not-git-ignored-files in the meantime or running the build-pipeline on a git-commit having a version-tag must not add, change or delete not-git-ignored-files. 8. A project must have a script update its dependencies. 9. Running the build-pipeline and running the script to update the dependencies must be runnable unattended. 10. The principles above must be implemented inside the repository and in a clean way. The rules regarding to running testcases, code-coverage and code-coverage-threshold are also applicable for codeunits which have testable source-code for obvious reasons. ## Usage The `CommonProjectStructure` is supposed to be used for software-projects for a better and consistent structure. A repository which implements the `CommonProjectStructure` can have multiple codeunits. It is recommended to bundle all codeunits of a repository which should always be released at the same time. ## Requirements ### General The repository must implement the requirements defined by [MinimalNormalizedContent](https://github.com/anionDev/ProjectTemplates/blob/main/Conventions/RepositoryStructure/MinimalNormalizedContent/MinimalNormalizedContent.md) with the following updates/changes: #### ReadMe.md The `ReadMe.md`-file must also contain: - A runtime- and development-dependency-list for all codeunits - Applied branching-system - Information if contribution is allowed/desired and, if yes, which constraints/conventions/conditions are associated with it - Information about which version of common project structure will be applied (define to support the "latest" version with the condition to update the repository if the requirements changes is also fine) - version of the applied common project structure-rules instead of the version of the applied `MinimalNormalizedContent`-rules Furthermore the repository must contain the following file with appropriate content: - [`GitVersion.yml`](https://github.com/GitTools/GitVersion) - [`.code-workspace`](https://code.visualstudio.com/docs/editor/workspaces) The `ReadMe.md`-file must also contain the "worst" [development-state](https://github.com/anionDev/ProjectTemplates/blob/other/update/Conventions/RepositoryStructure/CommonProjectStructure/CommonProjectStructure.md#readmemd-1) of all codeunits which are available in the project. #### security.txt The repository must contain a [`.well-known/security.txt`](https://securitytxt.org/)-file. #### Code-units A codeunit is a compilable part of the project. Small projects may often have only one codeunit, that is no problem. A typical modern web-application for example will probably have at least 2 codeunits (e. g. for web-part and backend-part). Testcases for a codeunit are a mandatory part of the codeunit. For each codeunit the repository must contain the following files and folder with appropriate content: - `/.codeunit.xml` - `/Other/QualityCheck/RunTestcases.py` - `/Other/QualityCheck/Linting.py` - `/Other/Reference/GenerateReference.py` - `/Other/Reference/ReferenceContent` - `/Other/CommonTasks.py` - `/Other/UpdateDependencies.py` - `/Other/OnBuildingFinished.py` - `/Other/Build/Build.py` - `/Other/Reference/ReferenceContent/Hints.md` - `/Other/ReadMe.md` All of these python-scripts are supposed to be executable on every developer-machine without any special software or sensitive data like keys which are typically not available on a development-machine. `` must be replaced by the name of the codeunit. There are the following rules for the name of a codeunit: - It is supposed to be a meaningful English name or abbreviation. - It must be in [Pascal-case](https://www.theserverside.com/definition/Pascal-case). - It is primary intended to be a project-internal label. A merge on the main-branch is only allowed if the scripts `CommonTasks.py`, `RunTestcases.py`, `Linting.py`, `GenerateReference.py` exits with 0 for each codeunit. It is also recommended to run `Build.py` for each codeunit to ensure that the build-script also runs without any errors. It is absolutely fine for a codeunit to have the same name like the project where it is contained in. ### Updates For every codeunit there must be at least one update be released every year. ### Dependencies #### Referenced packages Every release must use the latest version of all dependencies. Not-latest dependencies are only allowed to be used if there are special reasons which makes it impossible to update. Nevertheless using dependencies which are both not the latest version and older than one year is generally not allowed. #### Dependent codeunits In the source-code of a codeunit it is not allowed to directly include (means: referencing and using) the source-code of another codeunit in the same repository. If a codeunit depends on another codeunit then it must refer its artifacts from the `/Other/Resources/DependentCodeUnits`-folder. For a clear dependency-overview the names of all direct dependent codeunits must be listed in `.codeunit.xml`. A codeunit is not allowed to have uncommon external dependencies which do have at least one of the following properties: - The dependency is not available in the dependency-manager of the used programming-language. - The dependency is usually not available on a development-machine/build-server/production-server. - The dependency is defined outside the `/`-folder. There is one exception for the third property: A codeunit is allowed to load content from the `/Other/Resources`-folder. ## Further specifications ### Versioning The project's version and the version of every codeunit must always match the regex `^\d+\.\d+\.\d+$`. The project's version is defined by the output of `gitversion /showVariable MajorMinorPatch`. The version must be incremented due to the conditions described by [SemVerPractise](../../Versioning/SemVerPractise.md). ### Style All files for reference and explanations must be written in markdown. All markdown-files must match the style-requirements defined in [https://github.com/DavidAnson/markdownlint](github.com/DavidAnson/markdownlint). ### Secrets The project is not allowed to receive secrets as commandline-arguments. All kind of secrets must be passed via reference. This means that the application is allowed to receive a reference (link, file-path, etc.) where the application is able to find the secret at runtime. ### Specific files and folders #### `` It is expected that the folder `` contains the source-code of the codeunit including its testcases. ##### Codeunit-internal Conventions If there is no hard defined project-structure defined by the programming-language or other "external circumstances" then the following sub-file-structure is recommended: - `/`: Folder for the source-code - `/Tests`: Folder for the testcases - `/Resources`: Folder for all kind of resources like pictures, logos, and so on. - `/Resources/Translations`: Folder for ui-translations of the codeunit. - `/Other`: Folder for all other things. If the used programming language provides a specific sourcecode-structure recommended by the maintainer of the used programming-language/framework then it is recommended to follow these conventions. This recommendation still applies if even the recommendations are contrary to the other regulations described here. The purpose of this is that some programming languages enforce a more or less strictly defined structure. It is often advisable to follow them and sometimes nearly impossible to not follow them. But it is definitely impossible in practice to find two programming languages or frameworks which have exactly the same project-structure, style-guide, etcetera which makes it simply impossible to have a unique file-structure over the entire project/repository if you have at least 2 codeunits which require another programming-language (e.g. for frontend and backend). The aim of the common project structure is to be flexible and usable for many programming languages and frameworks, no matter what their requirements and recommendations say. Style-mixtures are inevitable when you use different programming languages and this is absolutely fine because on the one hand this is normal and no real problem in practice. And on the other hand codeunits should be independent and there is no requirement to adjust the structure and style of a codeunit to another not recommended structure and style. But the style of codeunits can (and is supposed to) be consistent inside itself over the entire codeunit. #### `CommonTasks.py` It is expected that the file `CommonTasks.py` is a python3-script which exits with a non-zero-exitcode if it fails. This script must to do things like update the version in `.codeunit.xml` and if required update the version in other files of the codeunit. This script does also have to do build dependent code units. `CommonTasks.py` is allowed to copy the artifacts of dependent codeunits to the `/Other/Resources/DependentCodeUnits`-folder. #### `UpdateDependencies.py` `UpdateDependencies.py` is extended to update dependencies of the codeunit to the latest available minor/fix-version. `UpdateDependencies.py` is allowed to be omitted if and only if the codeunit does not use any third-party-software whose reference including its version is defined in a committed file in the codeunit-folder. #### `RunTestcases.py` It is expected that the file `RunTestcases.py` is a python3-script which exits with a non-zero-exitcode if at least one testcase fails. This script also has to check the minimum test-coverage-threshold defined in `.codeunit.xml`. If something like compiling is required to run the testcases then this script must also do that. This script must generate an artifact called `TestCoverage` with a file called `TestCoverage.xml`. This script is supposed to generate an artifact called `TestCoverageReport`. `RunTestcases.py` is allowed to be omitted if and only if there is no testable source-code contained in the code-unit. This applies for example for codeunits whose only content is something like a `Dockerfile`-file. #### `Linting.py` It is expected that the file `Linting.py` is a python3-script which exits with a non-zero-exitcode if there is at least one linting-issue. #### `GenerateReference.py` It is expected that the file `GenerateReference.py` is a python3-script which generates or updates the reference based on `ReferenceContent` in `GeneratedReference` and exits with a non-zero-exitcode if at least one error occurs. #### `ReferenceContent` The folder `ReferenceContent` must contain a reference for the codeunit. The content of this folder is the source of the `GeneratedReference`-folder. The `ReferenceContent`-folder must be committed. #### `GeneratedReference` It is expected that after running `GenerateReference.py` the folder `GeneratedReference` contains an `index.html`-file (and possibly other files) containing a html-reference of the codeunit based on `ReferenceContent`. #### `OnBuildingFinished.py` `OnBuildingFinished.py` is an optional script. The intention is to be able to do things after all preceding scripts. An example is to be able to run tasks at the end when you have all available build-artifacts. This script is not intended to release or deployment artifacts. #### `Build.py` It is expected that the file `Build.py` is a python3-script which creates the build-artefact of the codeunit for productive usage and exits with a non-zero-exitcode if the build fails for any reason. This scripts is supposed to execute a full build-pipeline including all usual things (including security-checks, etc. but not including unit-tests). The build-artefact must be placed in a subfolder of the `Artifacts`-folder. Pushing the build-artifact to a package-registry/build-artifact-feed or something like this is not a task for this script. A codeunit can have multiple build-artifacts generated by `Build.py`. Each of these build-artifacts must be placed in a separate subfolder of the `Artifacts`-folder. The build-artifacts should only differ in their way of distribution. So if there is a project for example which has program-code which is usable both as library and as cli-program (for example wheel-files which also have entry-points) then this should be distributed as one codeunit. If this is not possible for some reason then it must be split in 2 codeunits: One for the core-library-project and one for the CLI-program which uses the library. #### `Artifacts` Artifacts are things generated by the scripts mentioned in this document. All artifacts must be placed in a subfolder of the `Artifacts`-folder. The name of the subfolder is treated as the name of the artifact. The folder `Artifacts` must be git-ignored. Common names for build-artifacts are: - `Reference`: For a html-reference - `TestCoverage`: Contains a cobertura-test-coverage-report called `TestCoverage.xml` - `TestCoverageReport`: For a html-test-coverage-report of the `TestCoverage`-artifact - `BuildResult_`: For build-result-files - `BOM`: Contains the `.sbom.xml`-file "TargetEnvironment" must describe the common artifact-usage-environment of the artifact. Recommended names are: - `BuildResult_Wheel`: For [Wheel](https://pypi.org/project/wheel/)-packages - `BuildResult_Deb`: For [Deb](https://en.wikipedia.org/wiki/Deb_(file_format))-packages - `BuildResult_NuGet`: For [NuGet](https://www.nuget.org/)-packages - `BuildResult_OCIImage`: For [OCI](https://opencontainers.org/)-compliant application-images - `BuildResult_WebApplication`: For Web-applications (contains typically the `dist`-folder of node-projects etc.) - `BuildResult_DotNet_`: For .NET-builds (where "`rid`" is a valid runtime-identifier described [here](https://learn.microsoft.com/en-us/dotnet/core/rid-catalog)) Artifacts are only allowed to be released immediately after a merge to the `stable`-branch. It is not allowed to do intermediate releases or releasing different artifacts of a codeunit with the same name and codeunit-version but based on different code-bases. #### `.codeunit.xml` It is expected that the folder `` contains the file `.codeunit.xml` with the following xml-content: ```xml CodeUnitName 1.0.0 80 John Doe john.doe@example.org ``` The values inside the xml-document must obviously be adapted. `.codeunit.xml` must always match [codeunit.xsd](./codeunit.xsd) While the project-version-specification is defined by [MinimalRequirements](./MinimalRequirements.md) a codeunit-version is independent of the project version but the rules to change codeunit-versions are the same as the rules for changing the project-version. So a codeunit-version can always be the same as the project-version but it does not have to. #### `Hints.md` `Hints.md` must contain a `Requirements`-section for developers which contains a list and explanations of all requirements which are necessary to run all scripts mentioned above successfully. `Hints.md` must also contain a `IDE`-section which describes a recommended IDE and optionally recommended IDE-settings for the codeunit. #### `ReadMe.md` `ReadMe.md` must contain general information about the code-unit. It must also include a development-state from the table below which describes the development-state of the code-unit. | State | description | |-|-| | ![Development-state](https://img.shields.io/badge/development--state-active%20development-brightgreen) | Updated versions will be released regularly. The updates contain dependency-updates, bugfixes and new features. | | ![Development-state](https://img.shields.io/badge/development--state-maintenance%20updates%20only-green) | Updated versions will be released regularly. The updates usually contain only dependency-updates and security-relevant bugfixes. | | ![Development-state](https://img.shields.io/badge/development--state-inactive-red) | No updates will be released with dependency-updates or security-relevant bugfixes. | If the codeunit does not have at least one release of updated artifacts since 1 year ago then it must be treated as inactive. ### Specific artifacts #### API-specification If the build-artifacts of a codeunit contains an API-server then `Build.py` must also produce a build-artifact `APISpecification` which contains the file `.api.json` which contains the [open API-specification](https://www.openapis.org) for the http-server. #### Bill of materials Every kind of build-process in `Build.py` which creates an `BuildResult_` artifact from the source-code must also generate a [SBOM](https://cyclonedx.org/capabilities/sbom/)-artifact called `BOM` which contains a file called`.sbom.xml`. ## Examples ### Demonstration Examples how to create codeunits for specific well known frameworks (Angular, .NET, etc.) are [here](https://github.com/anionDev/CommonProjectStructureExamples). ### Implementation An example of functions which can be used for the `CommonProjectStructure` are contained in the [ScriptCollection](https://github.com/anionDev/ScriptCollection/blob/main/ScriptCollection/ScriptCollection/TasksForCommonProjectStructure.py).