Ctrl+K

Developer Guide

Developer Guide#

Software#

  • VS Code

    • Settings Sync

      • Install ‘Settings Sync’ extension

      • Sync settings from public gist: 87b79ab2b6d3dc30fcd703f2fe02b421

    • jupyter

      • Add startup script from /docs/launch.py to jupyter startup location /Users/{username}/.ipython/profile_default/startup. This will import everything each time the ipython interactive window is launched.

  • Azure Data Studio

  • Github Desktop

    • Install and create a github account

    • Set global username and email:

      • git config --global user.name "Fake Name"

      • git config --global user.email email email@company.com

    • Create empty folder ‘~/github’

    • git clone https://github.com/jaymegordo/SMSEventLog.git

    • git commands

      • Guide

      • check current staged files: git status

      • ignore line endings (if project is worked on between mac and windows): git config --global core.autocrlf true

      • exit logs/messages: q

      • show history: git log --pretty=oneline

      • tag version: git tag -a 3.0.0 -m "this is the fist release version"

      • vscode github pannel > commit

      • cmd + shift + p > git push: follow tags

      • Create release

    • Files sometimes show as modified when repo opened (eg from onedrive) on new computer

      • If ALL files show changed, its usually just the filemode changed

      • git config core.fileMode false

  • Git Bash

    • Better terminal for windows

    • Copy bash.bashrc to ~/.bash_profile

      • Sets aliases

  • Python 3.9

    • Install Python (single user only)

    • Select “add python to PATH” during install options

    • open command prompt, check python installed correctly with python --version

    • Poetry

      • curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python - --preview

      • Need to add to path %appdata%\Python\Scripts

      • Init virtual environment with poetry lock to create poetry.lock

      • Download and sync all packages from poetry.lock with poetry install (run this at any time if things get out of sync or broken)

      • Use poetry as main package manager

      • When working in the cmd line, activate environment with poetry shell to have access to developer packages (eg twine)

      • print list of installed packages:

        • poetry show or

        • poetry run pip list

      • Create requirements.txt: poetry export -f requirements.txt --output requirements.txt --without-hashes

      • Update single package: poetry update <packagename>

      • Install from custom forked repo: poetry add git+https://github.com/jaymegordo/python-chromedriver-autoinstaller.git@master#egg=chromedriver-autoinstaller

        • #egg comes from name defined in setup.py, will be name of install

        • -e flag makes the package ‘editable’ and puts in into src

  • Microsoft Visual C++ Redistributable for Visual Studio 2019

    • NOTE only needed to install bsdiff4 from source

      • bsdiff4 has problem finding python.h during install

    • May also need to install Visual Studio Community 2019 (for python native build tools)

  • GTK3

    • https://doc.courtbouillon.org/weasyprint/stable/first_steps.html#installation

    • Needed for weasprint/cairo/etc

    • installs libcairo libraries

    • Dont need to add to path

  • Chocolatey

    • Package management for windows

    • Open PowerShell as administrator

      • Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

  • [Make]

    • Open git bash as administrator

    • choco install make

    • Allows using the make commands defined in Makefile

  • Install pyodbc

  • count lines of code in project: pygount --suffix=py --format=summary smseventlog

    Language   Files    %     Code     %     Comment    %   
---------  -----  ------  -----  ------  -------  ------
Python        46  100.00  10748  100.00     1699  100.00
---------  -----  ------  -----  ------  -------  ------
Sum total     46          10748             1699

GUI#

Deployment#

Build Process#

  1. Commit all changes to git, add comments etc (use github menu in vscode)

  2. Increment version with eg bumpversion patch --verbose (must be run inside active poetry shell)

  3. Push tags to git git push --tags (only pushes tags, increments version on github)

  4. git push > pushes all changes (could also use vscode menu to push)

    • Local testing - Build exe using custom build script poetry run python -m build_scripts.build (see pyinstaller below for more info)

    • Production - Build exe with pyupdater:

      • Note - dev needs to get the .env file with saved settings/passwords (ask jayme)

      • first cd build_scripts (location of the build scripts)

      • Mac: zsh build.sh true (true = upload after build is complete)

      • Win: build.bat true

  6. Push build package to s3 bucket with zsh upload.sh or upload.bat

Build Software#

  • bump2version

    • Use bumpversion to control version increments

    • creates a git tag before pushing

    • testing: bumpversion patch --dry-run --verbose --allow-dirty

    • usage: bumpversion [major | minor | patch | release | build]

      • bumpversion patch > will increment version from 3.0.1 > 3.0.2a0

      • bumpversion minor > will increment version from 3.0.1 > 3.1.0a0 > 3.1.0

  • PyUpdater

    • PyUpdater handles package build (wraps PyInstaller), signs code, uploads to amazon s3, and handles app update checks/downloads/restarts

    • AppUpdatesDemo

    • Build exe and deploy to amazon s3 (windows) - `

  • PyInstaller

    • In project dir (SMS) (windows), poetry run python -m build_scripts.build (runs custom build python file which calls PyInstaller with args)

    • Only use PyInstaller on its own to create a test build because PyUpdater auto zips everything.

    • This will package app and output files to /dist/[mac|win]

    • Build Issues

      • Files in frozen py files need to be replaced after version updates to their packages. They contain small changes eg try/except for getting a version number. The originals break the exe build. If directly replacing these files doesn’t work, may need to dig into code and redo fixes somehow.

      1. Selenium - no console window

        • packages\selenium\webdriver\common\service.py

        # replace this for no console window on windows, line 72 ish
# self.process = subprocess.Popen(cmd, env=self.env,
#                                 close_fds=platform.system() != 'Windows',
#                                 stdout=self.log_file,
#                                 stderr=self.log_file,
#                                 stdin=PIPE)

if any("hide_console" in arg for arg in self.command_line_args()):
    self.process = subprocess.Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE, creationflags=0x08000000)
else:
    self.process = subprocess.Popen(cmd, env=self.env, close_fds=platform.system() != 'Windows', stdout=self.log_file, stderr=self.log_file, stdin=PIPE)

      2. Pyupdater

        • /pyupdater/client/__init__.py ~ line 434 in _get_manifest_from_http()

        • Possibly temp, versions-win.gz doesn’t exist yet

        # version_files = [self.version_file, self.version_file_compat]
version_files = [self.version_file_compat]

    • Build Issues (resolved, ignore)

      1. Pandas.io.formats.style

        1. open [virtualenv]/Lib/site-packages/pandas/io/formats/style.py

        2. change line 141 from:

          template = env.get_template("html.tpl")

# to

import os
path = os.path.dirname(__file__) + "/templates/html.tpl"
with open(path) as fin:
    data = fin.read()
    template = env.from_string(data)

      2. Weasyprint / Cairosvg / Cairoccfi

        • VERSION fails to load (fix __init__.py in both)

        • weasyprint/__init__.py - Line 24 - add:

          try:
    VERSION = __version__ = (ROOT / 'VERSION').read_text().strip()
except:
    VERSION = __version__ = ''

      3. Matplotlib (seems to be resolved)

        • this isssue

        • change PyInstaller\hooks\hook-matplotlib.py to:

          datas = [(mpl_data_dir, "matplotlib/mpl-data")]

      4. Tinycss2 / cssselect2

        • related to weasyprint

        • add VERSION file at .../site-packages/tinycss2/VERSION and .../site-packages/cssselect2/VERSION

  • Code Signing

    • (mac only so far)

    • Sign - codesign --deep -s "SMS Event Log" "/Applications/SMS Event Log.app"

    • Verify - codesign -dv --verbose=4 "/Applications/SMS Event Log.app"

  • pypi

    • create .pypirc file at ‘c:\users\username’ (one time, stores api key for login)

    • make sure user installed setuptools is up to date (or bdist_wheel doesn’t work) python -m pip install --user --upgrade setuptools wheel

    • create wheel:

      • TODO: get version from github with requests?

      • python setup.py clean --pre && python setup.py sdist bdist_wheel clean --post

      • bdist_wheel > creates wheel (built distribution) to upload to pypi

      • sdist > creates source distribution (fallback)

      • clean –pre > runs custom command to remove previously leftover folders if they exist

      • clean –post > remove smseventlog.egg-info folder (never needed)

    • check contents of wheel package (optional) tar -tf dist\smseventlog-3.0.0.tar.gz

    • upload with twine

      • test twine upload --repository testpypi dist/*

      • live twine upload dist/*

    • cleanup all leftover folders: python setup.py clean --all

  • Azure Functions

    • Parts of the smseventlog package run as smseventlog-app in azure functions

    • This runs several functions such as: daily availability imports, oil sample imports, SMR hrs imports

    • Some key azure commands are:

      • Publish to azure func azure functionapp publish smseventlog-app --build-native-deps

      • Test locally func host start (must be run in active pipenv shell)

      • Manually trigger timer function curl --request POST -H "Content-Type:application/json" --data '{"input":""}' http://localhost:7071/admin/functions/az_TimerImportSMR

      • check subscriptions az account list --output table

      • change subscription az account set --subscription Jayme_Personal_Subscription

Contents