Development¶

Assumptions:

Docker and Docker-compose are installed and up-to-date
Python 3.6 or higher

Clone the repository:

$ git clone git@github.com:dougthor42/trendlines.git

Move into the directory, make a virutal environment, and activate it:

$ cd trendlines
$ python -m venv .venv
$ . .venv/bin/activate

Install requirements:

$ pip install -U pip setuptools wheel
$ pip install -r requirements.txt -r requirements-dev.txt
$ pre-commit install

Run tests:
```
$ pytest
```

Running with scripts¶

Make sure the config/localhost.cfg file exists:

# ./config/localhost.cfg
DEBUG = True
DATABASE = "./internal.db"
TRENDLINES_API_URL = "http://localhost:5000/api/v1/data"
broker_url = "redis://localhost"

In 3 separate shells, run these 3 commands in order

docker run -p 6379:6379 redis
python runworker.py
python runserver.py

You can also optionally run each in the background (-d for docker and & for the others), but personally I like to see the logs scroll by.

From a 4th shell, send data using the plaintext protocol:

echo "metric.name 12.345 `date +%s`" | nc localhost 2003

And view the data by opening http://localhost:5000 in your browser.

Running with Docker-Compose¶

The default configuration assumes running within docker-compose. If you need different settings, create config/trendlines.cfg and add your variables to it.

Build the images and bring up the stack:

$ docker-compose -f docker-build.yml build
$ docker-compose -f docker-build.yml up

Send data in

echo "metric.name 12.345 `date +%s`" | nc localhost 2003

And view the data by opening http://localhost:5000 in your browser.

Building the Docker Image¶

This is handled in CI, but in case you need to do it manually:

docker build -f docker/Dockerfile -t trendlines:latest -t dougthor42/trendlines:latest .
docker push dougthor42/trendlines:latest

Database Migrations¶

This project uses peewee-moves to handle migrations. The documentation for that project is a little lacking, but I found it a litte easier to use than the more-popular peewee-migrate. peewee-moves also has more documentation.

To apply migrations to an exsiting database that has never had any migrations applied:

Open the database.

Manually create the following table (adjust syntax accordingly):

CREATE TABLE migration_history (
  `id` INT NOT NULL AUTO_INCREMENT,
  `name` VARCHAR(255) NOT NULL,
  `date_applied` DATETIME NOT NULL,
PRIMARY_KEY (`id`));

Populate the table with all the migrations that have already been applied. The name value should match the migration filename, sans .py extension, and the date_applied field can be any timestamp.

INSERT INTO `migration_history`
  (`name`, `date_applied`)
  VALUES
    ('0001_create_table_metric', '2019-02-14 14:56:37'),
    ('0002_create_table_datapoint', '2019-02-14 14:56:37');

Verify that things are working. You should see [x] for all migrations:

$ peewee-db --directory migrations --database sqlite:///internal.db status
INFO: [x] 0001_create_table_metric
INFO: [x] 0002_create_table_datapoint

Creating a New Table¶

Create the table in trendlines.orm.

Create the new table migration:

$ peewee-db --directory migrations \
            --database sqlite:///internal.db \
            create \
            trendlines.orm.NewTable

And then apply it:

$ peewee-db --directory migrations \
            --database sqlite:///internal.db \
            upgrade

If you’re using the python shell, run the following for for step 3:

>>> from peewee import SqliteDatabase
>>> from peewee_moves import DatabaseManager
>>> manager = DatabaseManager(SqliteDatabase('internal.db')
>>> manager.create('trendlines.orm')
>>> manager.upgrade()

Modifying a Table¶

Modify the table in trendlines.orm.

Create the migration file:

$ peewee-db --directory migrations \
            --database sqlite:///internal.db \
            revision
            "short_revision_description spaces OK but not recommended"

Manually modify the upgrade and downgrade scripts in the new migration file.

Apply the migration:

$ peewee-db --directory migrations \
            --database sqlite:///internal.db \
            upgrade