Gitiles
Code Review Sign In
gerrit.avm99963.com / monorail-avm99963 / f779fa013a7948106b73568525602c66c658d734
commitf779fa013a7948106b73568525602c66c658d734[log] [tgz]
authorAdrià Vilanova Martínez <me@avm99963.com>Sun Apr 03 21:01:25 2022 +0200
committergerritbot <gerritbot@avm99963.com>Sun Apr 03 21:21:02 2022 +0200
tree6486dea9f692d07db0ef12cdb39ba5c0460c627f
parent4236530e8fce05ae5a4f1134cfb1803439a9f01e [diff]
Try to fix App Engine costs rising lately

Bug: misc:55
GitOrigin-RevId: fd8a9fc76e96c018c5e922a754d01abf3a0bcc7e
1 file changed
tree: 6486dea9f692d07db0ef12cdb39ba5c0460c627f
  1. api/
  2. businesslogic/
  3. doc/
  4. features/
  5. framework/
  6. google/
  7. project/
  8. proto/
  9. schema/
  10. search/
  11. services/
  12. sitewide/
  13. static/
  14. static_src/
  15. templates/
  16. testing/
  17. third_party/
  18. tools/
  19. tracker/
  20. components ⇨ ../../luci/appengine/components/components
  21. gae.py ⇨ ../../../infra/luci/appengine/components/tools/gae.py
  22. gae_ts_mon ⇨ ../../../infra/appengine_module/gae_ts_mon
  23. testing_utils ⇨ ../../../infra/appengine_module/testing_utils/
  24. .babelrc
  25. .coveragerc
  26. .eslintrc.json
  27. .expect_tests.cfg
  28. .expect_tests_pretest.py
  29. .gitignore
  30. .prettierrc.json
  31. .style.yapf
  32. .testcoveragerc
  33. app.yaml
  34. appengine_config.py
  35. codereview.settings
  36. cron.yaml
  37. dev-services.yml
  38. jsconfig.json
  39. karma.conf.js
  40. Makefile
  41. monorailapp.py
  42. OWNERS
  43. package-lock.json
  44. package.json
  45. PRESUBMIT.py
  46. queue.yaml
  47. README.md
  48. registerpages.py
  49. requirements.dev.txt
  50. requirements.py2.txt
  51. requirements.txt
  52. settings.py
  53. tsconfig.json
  54. webpack.config.js
README.md

Monorail Issue Tracker - bugs.avm99963.com

This repo is a fork of the open-source Monorail issue tracker designed to run at bugs.avm99963.com. I've made the code public so it can inspire other developers to host relatively cheap Monorail instances.

Monorail development should generally be contributed upstream, this repo will only accept changes specific to small and relatively cheap instances.

The original README.md file is here below (slightly modified where applicable):

Monorail Issue Tracker

Monorail is the Issue Tracker used by the Chromium project and other related projects. It is hosted at bugs.chromium.org.

If you wish to file a bug against Monorail itself, please do so in our self-hosting tracker. We also discuss development of Monorail at infra-dev@chromium.org.

Getting started with Monorail development

For Googlers: Monorail's codebase is open source and can be installed locally on your workstation of choice.

Here's how to run Monorail locally for development on MacOS and Debian stretch/buster or its derivatives.

  1. You need to get the Chrome Infra depot_tools commands to check out the source code and all its related dependencies and to be able to send changes for review.

  2. Check out the Monorail source code

    1. cd /path/to/empty/workdir
    2. fetch infra (make sure you are not "fetch internal_infra" )
    3. cd infra/appengine/monorail

  3. Make sure you have the AppEngine SDK:

    1. It should be fetched for you by step 1 above (during runhooks)
    2. Otherwise, you can download it from https://developers.google.com/appengine/downloads#Google_App_Engine_SDK_for_Python
    3. Also follow https://cloud.google.com/appengine/docs/standard/python3/setting-up-environment to setup gcloud

  4. Spin up dependent services.

    1. We use docker and docker-compose to orchestrate. So install docker and docker-compose first. For glinux users see go/docker
    2. Run docker-compose -f dev-services.yml up -d. This should spin up:
      1. MySQL v5.6
      2. Redis
      3. Cloud Tasks Emulator

  5. Set up SQL database. (You can keep the same sharding options in settings.py that you have configured for production.).

    1. Copy setup schema into the docker container
      1. docker cp schema/. mysql:/schema
      2. docker exec -it mysql bash
      3. mysql --user=root monorail < schema/framework.sql
      4. mysql --user=root monorail < schema/project.sql
      5. mysql --user=root monorail < schema/tracker.sql
      6. exit

  6. Configure the site defaults in settings.py. You can leave it as-is for now.

  7. Set up the front-end development environment:

    1. On Debian
      1. eval `../../go/env.py` -- you'll need to run this in any shell you wish to use for developing Monorail. It will add some key directories to your $PATH.
      2. Install build requirements:
        1. sudo apt-get install build-essential automake
    2. On MacOS
      1. Install homebrew
      2. Install node and npm

        1. Install node version manager brew install nvm

        2. See the brew instructions on updating your shell's configuration

        3. Install node and npm nvm install 12.13.0

        4. Add the following to the end of your ~/.zshrc file:

          export NVM_DIR="$HOME/.nvm"
[ -s "/usr/local/opt/nvm/nvm.sh" ] && . "/usr/local/opt/nvm/nvm.sh"  # This loads nvm
[ -s "/usr/local/opt/nvm/etc/bash_completion.d/nvm" ] && . "/usr/local/opt/nvm/etc/bash_completion.d/nvm"  # This loads nvm bash_completion

  8. Install Python and JS dependencies:

    1. Install MySQL, needed for mysqlclient
      1. For mac: brew install mysql@5.6
      2. For Debian derivatives, download and unpack this bundle: tar -xf mysql-server_5.6.40-1ubuntu14.04_amd64.deb-bundle.tar. Install the packages in the order of mysql-common,mysql-community-client, mysql-client, then mysql-community-server.
    2. Optional: You may need to install pip. You can verify whether you have it installed with which pip.
      1. make sure to install pip using python2 instead of python3 (use python --version to check the version, which python2 to check the path)
      2. curl -O https://bootstrap.pypa.io/pip/2.7/get-pip.py
      3. sudo python get-pip.py
    3. Use virtualenv to keep from modifying system dependencies.
      1. sudo pip install virtualenv
      2. virtualenv venv to set up virtualenv within your monorail directory.
      3. source venv/bin/activate to activate it, needed in each terminal instance of the directory.
    4. Mac only: install libssl, needed for mysqlclient. (do this in local env not virtual env)
      1. brew install openssl; export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/opt/openssl/lib/
    5. make dev_deps (run in virtual env)
    6. make deps (run in virtual env)

  9. Run the app:

    1. make serve (run in virtual env)

  10. Browse the app at localhost:8080 your browser.

  11. Set up your test user account (these steps are a little odd, but just roll with it): 1. Sign in using test@example.com 1. Log back out and log in again as example@example.com 1. Log out and finally log in again as test@example.com. 1. Everything should work fine now.

  12. Modify your Monorail User row in the database and make that user a site admin. You will need to be a site admin to gain access to create projects through the UI.

    1. docker exec mysql mysql --user=root monorail -e "UPDATE User SET is_site_admin = TRUE WHERE email = 'test@example.com';"
    2. If the admin change isn't immediately apparent, you may need to restart your local dev appserver. If you kill the dev server before running the docker command, the restart may not be necessary.

Instructions for deploying Monorail to an existing instance or setting up a new instance are here.

Feature Launch Tracking

To set up FLT/Approvals in Monorail:

  1. Visit the gear > Development Process > Labels and fields
  2. Add at least one custom field with type "Approval" (this will be your approval
  3. Visit gear > Development Process > Templates
  4. Check "Include Gates and Approval Tasks in issue"
  5. Fill out the chart - The top row is the gates/phases on your FLT issue and you can select radio buttons for which gate each approval goes

Testing

To run all Python unit tests, in the appengine/monorail directory run:

make test

For quick debugging, if you need to run just one test you can do the following. For instance for the test IssueServiceTest.testUpdateIssues_Normal in services/test/issue_svc_test.py:

../../test.py test appengine/monorail:services.test.issue_svc_test.IssueServiceTest.testUpdateIssues_Normal --no-coverage

Frontend testing

To run the frontend tests for Monorail, you first need to set up your Go environment. From the Monorail directory, run:

eval `../../go/env.py`

Then, to run the frontend tests, run:

make karma

If you want to skip the coverage for karma, run:

make karma_debug

To run only one test or a subset of tests, you can add .only to the test function you want to isolate:

// Run one test.
it.only(() => {
  ...
});

// Run a subset of tests.
describe.only(() => {
  ...
});

Just remember to remove them before you upload your CL.

Troubleshooting

  • BindError: Unable to bind localhost:8080

This error occurs when port 8080 is already being used by an existing process. Oftentimes, this is a leftover Monorail devserver process from a past run. To quit whatever process is on port 8080, you can run kill $(lsof -ti:8080).

  • RuntimeError: maximum recursion depth exceeded while calling a Python object

If running make serve gives an output similar to this,

  1. make sure you're using a virtual environment (see above for how to configure one). Then, make the changes outlined in this CL.
  2. Also try pip install protobuf
  • gcloud: command not found

Add the following to your ~/.zshrc file: alias gcloud='/Users/username/google-cloud-sdk/bin/gcloud'. Replace username with your Google username.

  • TypeError: connect() got an unexpected keyword argument 'charset'

This error occurs when dev_appserver cannot find the MySQLdb library. Try installing it via sudo apt-get install python-mysqldb.

  • TypeError: connect() argument 6 must be string, not None

This occurs when your mysql server is not running. Check if it is running with ps aux | grep mysqld. Start it up with /etc/init.d/mysqld start on linux, or just mysqld.

  • dev_appserver says OSError: [Errno 24] Too many open files and then lists out all source files

dev_appserver wants to reload source files that you have changed in the editor, however that feature does not seem to work well with multiple GAE modules and instances running in different processes. The workaround is to control-C or kill the dev_appserver processes and restart them.

  • IntegrityError: (1364, "Field 'comment_id' doesn't have a default value") happens when trying to file or update an issue

In some versions of SQL, the STRICT_TRANS_TABLES option is set by default. You'll have to disable this option to stop this error.

  • ImportError: No module named six.moves

You may not have six.moves in your virtual environment and you may need to install it.

  1. Determine that python and pip versions are possibly in vpython-root
    1. which python
    2. which pip
  2. If your python and pip are in vpython-root
    1. sudo `which python` `which pip` install six
  • enum hash not match when run make dev_peds

You may run the app using python3 instead of python2.

  1. Determine the python version used in virtual environment python --version if it's 3.X

    deactivate

    rm -r venv/

    pip uninstall virtualenv

    pip uninstall pip

    in local environment python --version make sure to change it to python2

    follow previous to instructions to reinstall pip and virtualenv

  • mysql_config not found when run make dev_deps

    this may be caused installing the wrong version of six. run pip list in virtual env make sure it is 1.15.0 if not

    deactivate

    rm -r venv/

    pip uninstall six

    pip install six==1.15.0

    virtualenv venv

Development resources

Supported browsers

Monorail supports all browsers defined in the Chrome Ops guidelines.

File a browser compatability bug here.

Frontend code practices

See: Monorail Frontend Code Practices

Monorail's design

Triage process

See: Monorail Triage Guide.

Release process

See: Monorail Deployment

User guide

For information on how to use Monorail, see the Monorail User Guide.

Setting up a new instance of Monorail

See: Creating a new Monorail instance