Deployment Process

5.1 Determine the processes during deployment

Describe all processes that are running during deployment: Build, deploy and post-deploy phases. Explain why downtime occurs on your project

Build and deploy full steps:

Code and configuration validation
- run series of checks and code validation
- modify cluster topology (e.g. elasticsearch was added)
- run composer install
- on syntax error in a configuration file, Git server refuses the push, see Protective block
  - to disable protective block
```
# .magento.app.yaml
preflight:
   enabled: false
```
Build
- site is not in maintenance mode
- will not be brought down if errors or issues occur
- cluster has not been created yet (don't try to connect to a database or assume anything was daemonized)
- execute commands before packaging your application
- runs hooks in the build section from .magento.app.yaml
```
# We run build hooks before your application has been packaged.
build: |
    set -e
    php ./vendor/bin/ece-tools build:generate
    php ./vendor/bin/ece-tools build:transfer
   # old style
   #php ./vendor/bin/ece-tools build
```
- result of the build phase is a read-only file system referred to as a slug

Prepare the slug

create an archive and put the slug in permanent storage
slug includes all files and folders excluding the mounts configured in magento.app.yaml

mounts:
    "var": "shared:files/var"
    "app/etc": "shared:files/etc"
    "pub/media": "shared:files/media"
    "pub/static": "shared:files/static"

Deploy slugs and cluster
- file systems are read-only
- mounts each service in a container (web server, Elasticsearch, RabbitMQ)
- mounts the read-write file system (mounted on a highly available distributed storage grid)
- configures the network so services can “see” each other (and only each other)
Deployment hooks
- freeze the incoming traffic at the entry point for 60 seconds
- puts the application in maintenance mode until deployment is complete
- execute commands after packaging and deploying your application
- runs hooks in the deploy section from .magento.app.yaml
```
# We run deploy hook after your application has been deployed and started.
deploy: |
    php ./vendor/bin/ece-tools deploy
```
- deploy script uses the values defined by configuration files in the .magento directory, then the script deletes the directory and its contents
Note:

During the deployment process, all connections queue for up to 5 minutes preserving any active sessions and pending actions, such as adding to cart or checkout. After deployment, the queue is released and connections continue without interruption. To use this connection hold to your advantage and reduce downtime to zero, you must configure your project to use the most efficient deploy strategy.
Post-deployment: configure routing
- creates backup (BAK) files for the app/etc/env.php and the app/etc/config.php configuration files
- execute commands after deploying your application and after the container begins accepting connections
- runs hooks in the post_deploy section from .magento.app.yaml
```
# We run post deploy hook to clean and warm the cache. Available with ECE-Tools 2002.0.10.
post_deploy: |
    php ./vendor/bin/ece-tools post-deploy
```

What role every process/phase plays and how to impact every process

How Magento Cloud deploys Magento. What every script does on every deployment phase

Ece-tools \Magento\MagentoCloud\App\Container initialize \Illuminate\Container\Container and configure all commands processes.

Build

php ./vendor/bin/ece-tools build is a proxy for calling build:generate and build:transfer commands.

    public function execute(InputInterface $input, OutputInterface $output)
    {
        $this->getApplication()->find(\Magento\MagentoCloud\Command\Build\Generate::NAME)->execute($input, $output);
        $this->getApplication()->find(\Magento\MagentoCloud\Command\Build\Transfer::NAME)->execute($input, $output);
    }

php ./vendor/bin/ece-tools build:generate

notice: Starting generate command. (magento/ece-tools version: 2002.0.17, magento/magento2-base version: 2.2.8)

Process\Build\PreBuild
- check VERBOSE_COMMANDS in .magento.env.yaml
- clean generated/code folder
- clean generated/metadata folder
- delete flag .static_content_deploy file

Process\ValidateConfiguration

    $this->container->make(\Magento\MagentoCloud\Process\ValidateConfiguration::class, [
        'validators' => [
            ValidatorInterface::LEVEL_CRITICAL => [
                $this->container->make(ConfigValidator\Build\ComposerFile::class),
                $this->container->make(ConfigValidator\Build\StageConfig::class),
                $this->container->make(ConfigValidator\Build\BuildOptionsIni::class),
            ],
            ValidatorInterface::LEVEL_WARNING => [
                $this->container->make(ConfigValidator\Build\ConfigFileExists::class),
                $this->container->make(ConfigValidator\Build\DeprecatedBuildOptionsIni::class),
                $this->container->make(ConfigValidator\Build\StageConfigDeprecatedVariables::class),
                $this->container->make(ConfigValidator\Build\ModulesExists::class),
                $this->container->make(ConfigValidator\Build\AppropriateVersion::class),
                $this->container->make(ConfigValidator\Build\ScdOptionsIgnorance::class),
                $this->container->make(ConfigValidator\IdealState::class),
            ],
        ],
    ]),

validators list: Config/Validator/Build/

Process\Build\RefreshModules
- see \Config\Module::refresh
- enable all modules php ./bin/magento module:enable --all --ansi --no-interaction if modules not found in the app/etc/config.php
- do not enable already disabled modules (if modules found in the app/etc/config.php)
Process\Build\ApplyPatches
- applying patches (Patch\Manager::applyAll)
  - copyStaticFile: copy pub/static.php => pub/front-static.php
  - applyComposerPatches: from patches.json file
  - applyHotFixes: git apply patches from m2-hotfixes/*.patch
Process\Build\MarshallFiles
- delete var/cache/ directory
- copying di.xml files for Magento version < 2.2
Process\Build\CopySampleData
- copy (if exists) vendor/magento/sample-data-media => /pub/media
Process\Build\CompileDi
- execute php ./bin/magento setup:di:compile {$verbosityLevel} --ansi --no-interaction
Process\Build\ComposerDumpAutoload
- execute composer dump-autoload -o --ansi --no-interaction
Process\Build\DeployStaticContent
- delete .static_content_deploy flag file
- validate Config\Validator\GlobalStage\ScdOnBuild::validate
  - check SCD_ON_DEMAND: false
  - check SKIP_SCD: false
  - validate Config\Validator\Build\ConfigFileStructure::validate
    - in the app/etc/config.php
      - scopes/websites (count($websites) > 0)
      - scopes/stores (count($stores) > 0 )
- if all checks valid, execute Process\Build\DeployStaticContent\Generate
  - build SCD commands StaticContent\CommandFactory::matrix
    - StaticContent\CommandFactory::build
      - command: php ./bin/magento setup:static-content:deploy --ansi --no-interaction
      - add -f option (for magento > 2.2)
      - add -s {$stratagy} from SCD_STRATEGY from .magento.env.yaml (for magento > 2.2)
      - add verbosity level (-v, -vv, -vvv) from VERBOSE_COMMANDS from .magento.env.yaml
      - add --jobs {$threadCount} threads count: SCD_THREADS from .magento.env.yaml
      - add --no-html-minify by SKIP_HTML_MINIFICATION (default: true) from .magento.env.yaml
      - result example: php ./bin/magento setup:static-content:deploy --ansi --no-interaction -f --jobs 3 --no-html-minify
    - default command (StaticContent\CommandFactory::create($option, array_keys($matrix))) based on given options
      - run: StaticContent\CommandFactory::build
      - get excluded themes: SCD_EXCLUDE_THEMES from .magento.env.yaml (deprecated)
      - exclude themes from SCD_MATRIX
      - add unique locales StaticContent\Build\Option::getLocales
        
        ADMIN_LOCALE from (MAGENTO_CLOUD_VARIABLES) (default: en_US)
        
        general/locale/code from app/etc/config.php
        
        admin_user/locale/code from app/etc/config.php
        
        add en_US
      - result example: php ./bin/magento setup:static-content:deploy --ansi --no-interaction -f --jobs 3 --no-html-minify --exclude-theme Magento/blank --exclude-theme Magento/luma en_US nb_NO
    - generate commands based on SCD_MATRIX from .magento.env.yaml
      - run: StaticContent\CommandFactory::build
      - add --theme {$resolvedTheme}
      - add language from SCD_MATRIX (en_US, nb_NO)
      - result example: php ./bin/magento setup:static-content:deploy --ansi --no-interaction -f --jobs 3 --no-html-minify --theme Magento/blank en_US nb_NO
    - execute all generated scd commands
- save .static_content_deploy flag file

notice: Generate command completed.

php ./vendor/bin/ece-tools build:transfer

notice: Starting transfer files.

Process\Build\CompressStaticContent
- check .static_content_deploy flag file
- Util\StaticContentCompressor::process
  - get gzip compression level in SCD_COMPRESSION_LEVEL (default: 6 (build stage) or 4 (deploy stage)) from .magento.env.yaml
  - get seconds maximum time for running static compression command in SCD_COMPRESSION_TIMEOUT (default: 600) from .magento.env.yaml
  - get VERBOSE_COMMANDS from .magento.env.yaml
  - command result: /usr/bin/timeout -k 30 600 /bin/bash -c 'find '\''/var/magento/pub/static'\'' -type d -name '\''DELETING_*'\'' -prune -o -type f -size +300c '\''('\'' -name '\''*.js'\'' -or -name '\''*.css'\'' -or -name '\''*.svg'\'' -or -name '\''*.html'\'' -or -name '\''*.htm'\'' '\'')'\'' -print0 | xargs -0 -n100 -P16 gzip -q --keep -6'
Process\Build\ClearInitDirectory
- clear temporary directory: ./init/
- delete app/etc/env.php file
Process\Build\BackupData
- Process\Build\BackupData\StaticContent
  - delete var/.regenerate flag file
  - check .static_content_deploy flag file
  - clear ./init/pub/static/ folder
  - move pub/static/ to init/pub/static
- Process\Build\BackupData\WritableDirectories
  - copy app/etc/ to init/app/etc/
  - copy pub/media/ to init/pub/media/
  - copy var/view_preprocessed/ to init/var/view_preprocessed/ if SKIP_HTML_MINIFICATION = false (Default: true)
  - copy var/log to init/var/log (e.g. cloud.log file)

notice: Transfer completed.

Deploy

php ./vendor/bin/ece-tools deploy

notice: Starting deploy. (magento/ece-tools version: 2002.0.17, magento/magento2-base version: 2.2.8)

delete var/.deploy_is_failed flag file
Process\Deploy\PreDeploy
- Runs all processes that have to be run before deploy starting. see Process\Deploy\PreDeploy folders
  - Process\PreDeploy\ConfigUpdate\Cache
    - CACHE_CONFIGURATION from .magento.env.yaml
  - Process\Deploy\PreDeploy\CleanStaticContent
    - check .static_content_deploy flag file exists
    - CLEAN_STATIC_FILES is true
    - clean pub/static/ folder
  - Process\Deploy\PreDeploy\CleanViewPreprocessed
    - clean var/view_preprocessed directory when the deployment variable SKIP_HTML_MINIFICATION is true
  - Process\Deploy\PreDeploy\CleanRedisCache
    - redis-cli -h $redisHost -p $redisPort -n $redisCacheDb flushdb ($redisCacheDb = 1)
  - Process\Deploy\PreDeploy\CleanFileCache
    - delete /var/cache folder
  - Process\Deploy\PreDeploy\RestoreWritableDirectories
    - see Filesystem\RecoverableDirectoryList::getList
    - copy init/app/etc to app/etc
    - copy init/pub/media to pub/media
    - copy (symlink) init/pub/static to pub/static
    - delete var/.regenerate flag file
  - Process\Deploy\PreDeploy\SetProductionMode
    - switching magento to production mode (app/etc/env.php => 'MAGE_MODE' => 'production',)
- Enabling Maintenance mode: php ./bin/magento maintenance:enable --ansi --no-interaction
Process\Deploy\DisableCron
- disable cron (app/etc/env.php => ['cron' => ['enabled' => 0]])

Process\ValidateConfiguration

    $this->container->make(\Magento\MagentoCloud\Process\ValidateConfiguration::class, [
    'validators' => [
        ValidatorInterface::LEVEL_CRITICAL => [
            $this->container->make(ConfigValidator\Deploy\DatabaseConfiguration::class),
            $this->container->make(ConfigValidator\Deploy\ResourceConfiguration::class),
            $this->container->make(ConfigValidator\Deploy\SessionConfiguration::class),
        ],
        ValidatorInterface::LEVEL_WARNING => [
            $this->container->make(ConfigValidator\Deploy\AdminData::class),
            $this->container->make(ConfigValidator\Deploy\PhpVersion::class),
            $this->container->make(ConfigValidator\Deploy\SearchEngine::class),
            $this->container->make(ConfigValidator\Deploy\ElasticSearchUsage::class),
            $this->container->make(ConfigValidator\Deploy\ElasticSearchVersion::class),
            $this->container->make(ConfigValidator\Deploy\AppropriateVersion::class),
            $this->container->make(ConfigValidator\Deploy\ScdOptionsIgnorance::class),
            $this->container->make(ConfigValidator\Deploy\DeprecatedVariables::class),
            $this->container->make(ConfigValidator\Deploy\RawEnvVariable::class),
            $this->container->make(ConfigValidator\Deploy\MagentoCloudVariables::class),
            $this->container->make(ConfigValidator\Deploy\JsonFormatVariable::class),
        ],
    ],
]),

validators list: Config/Validator/Deploy/

Process\Deploy\UnlockCronJobs
- In magento version 2.2 was implemented locking functionality for cron jobs, new cron jobs can't be started if exist job in status 'running' with same 'job_code'.
- UPDATE `cron_schedule` SET `status` = 'error', `messages` = 'The job is terminated due to system upgrade' WHERE `status` = 'running'
Process\Deploy\SetCryptKey
- update crypt/key in app/etc/env.php with CRYPT_KEY variable value (MAGENTO_CLOUD_VARIABLES)

Process\Deploy\InstallUpdate

magento not installed
- see Process/Deploy/InstallUpdate/Install
- Process\Deploy\InstallUpdate\Install\Setup: php ./bin/magento setup:install -n --session-save=db --cleanup-database
- run php ./bin/magento app:config:import --ansi --no-interaction
- sends email with link to reset password
magento already installed
- see Process/Deploy/InstallUpdate/Update
- sets an admin URL from ADMIN_URL variable value (MAGENTO_CLOUD_VARIABLES) (Process\Deploy\InstallUpdate\Update\SetAdminUrl)
- Running setup upgrade:php ./bin/magento setup:upgrade --keep-generated --ansi --no-interaction (Process\Deploy\InstallUpdate\Update\Setup)

Updating configuration from environment variables (DeployProcess\InstallUpdate\ConfigUpdate)

see Process/Deploy/InstallUpdate/ConfigUpdate

    $this->container->make(DeployProcess\InstallUpdate\ConfigUpdate\PrepareConfig::class),
    $this->container->make(DeployProcess\InstallUpdate\ConfigUpdate\CronConsumersRunner::class),
    $this->container->make(DeployProcess\InstallUpdate\ConfigUpdate\DbConnection::class),
    $this->container->make(DeployProcess\InstallUpdate\ConfigUpdate\Amqp::class),
    $this->container->make(DeployProcess\InstallUpdate\ConfigUpdate\Session::class),
    $this->container->make(DeployProcess\InstallUpdate\ConfigUpdate\SearchEngine::class),
    $this->container->make(DeployProcess\InstallUpdate\ConfigUpdate\Urls::class),
    $this->container->make(DeployProcess\InstallUpdate\ConfigUpdate\DocumentRoot::class),

  $this->container->when(DeployProcess\InstallUpdate\ConfigUpdate\Urls::class)
  ->needs(ProcessInterface::class)
  ->give(function () {
      return $this->container->makeWith(ProcessComposite::class, [
          'processes' => [
              $this->container->make(DeployProcess\InstallUpdate\ConfigUpdate\Urls\Database::class),
              $this->container->make(DeployProcess\InstallUpdate\ConfigUpdate\Urls\Environment::class),
          ],
      ]);
  });

Process\Deploy\DeployStaticContent
- clean pub/static and var/view_preprocessed if SCD_ON_DEMAND = true
- skip this step if SKIP_SCD = true or .static_content_deploy file flag exists
- clean pub/static and var/view_preprocessed if CLEAN_STATIC_FILES = true
- Generating fresh static content
  - see Process\Deploy\DeployStaticContent\Generate
  - see build SCD commands StaticContent\CommandFactory::matrix
Process\Deploy\CompressStaticContent
- skip if SCD_ON_DEMAND = true
- skip if exists .static_content_deploy flag
- use Util\StaticContentCompressor::process
Process\Deploy\DisableGoogleAnalytics
- if ENABLE_GOOGLE_ANALYTICS = false (default: false) in the .magento.env.yaml
- if not master branch (/^(master|production|staging)(?:-[a-z0-9]+)?$/i) (check MAGENTO_CLOUD_ENVIRONMENT env variable)
- execute UPDATE `core_config_data` SET `value` = 0 WHERE `path` = 'google/analytics/active'

Process\Deploy\DeployCompletion

checks is post_deploy hook enabled in .magento.app.yaml
runs processes if only post_deploy hook is not configured

$this->container->when(DeployProcess\DeployCompletion::class)
    ->needs(ProcessInterface::class)
    ->give(function () {
        return $this->container->makeWith(ProcessComposite::class, [
            'processes' => [
                $this->container->make(PostDeployProcess\EnableCron::class),
                $this->container->make(PostDeployProcess\Backup::class),
                $this->container->make(PostDeployProcess\CleanCache::class),
            ],
        ]);
    });

Disable maintenance mode: php ./bin/magento maintenance:disable --ansi --no-interaction

notice: Deployment completed.

Post_deploy

php ./vendor/bin/ece-tools post-deploy

notice: Starting post-deploy.

check var/.deploy_is_failed flag file and skip post-deploy
Config\Validator\Deploy\DebugLogging
- check dev/debug/debug_logging is disabled in production environments
Process\PostDeploy\EnableCron
- enable cron (app/etc/env.php => unset($config['cron']['enabled']);)
Process\PostDeploy\Backup
- \Magento\MagentoCloud\Filesystem\BackupList::getList
- copy app/etc/env.php to app/etc/env.php.bak
- copy app/etc/config.php to app/etc/env.php.bak
Process\PostDeploy\CleanCache
- execute: php ./bin/magento cache:flush --ansi --no-interaction
Process\PostDeploy\WarmUp
- Process\PostDeploy\WarmUp::getUrlsForWarmUp
- call async urls from WARM_UP_PAGES in the .magento.env.yaml
- use web/unsecure/base_url and web/secure/base_url from core_config_data table

notice: Post-deploy is complete.

How to extend these scripts and best practices for doing so

Describe the ways to retrieve logs for phases and its scripts

Build logs

Logs from the build hook are redirected to the output stream of git push.

var/log/cloud.log file inside the Magento application root directory, compiles build, deploy, post-deploy actions into one file.

Deploy logs

var/log/cloud.log file inside the Magento application root directory, compiles build, deploy, post-deploy actions into one file.

Logs from the deploy hook are located on the server in the following locations:

Integration: /var/log/deploy.log
Staging: /var/log/platform/_stg/deploy.log
Production: /var/log/platform//deploy.log

Post Deploy logs

var/log/cloud.log file inside the Magento application root directory, compiles build, deploy, post-deploy actions into one file.

Logs from the post-deploy hook are located on the server in the following locations:

Integration: /var/log/post-deploy.log
Staging: /var/log/platform/_stg/post_deploy.log
Production: /var/log/platform//post_deploy.log

Note1: var/log/cloud.log logs on the Staging and Production environments are only available on the first node in the cluster.

Note2: Logs in the /var/log/ directory are not shared between nodes of the enterprise server cluster; each server has its own log. We recommend checking logs on every node for complete reporting.

5.2 Demonstrate the ability to create Magento Cloud script configurations

Documentation:

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deployment Process

5.1 Determine the processes during deployment

Describe all processes that are running during deployment: Build, deploy and post-deploy phases. Explain why downtime occurs on your project

What role every process/phase plays and how to impact every process

How Magento Cloud deploys Magento. What every script does on every deployment phase

How to extend these scripts and best practices for doing so

Describe the ways to retrieve logs for phases and its scripts

5.2 Demonstrate the ability to create Magento Cloud script configurations

FilesExpand file tree

5. Deployment Process.md

Latest commit

History

5. Deployment Process.md

File metadata and controls

Deployment Process

5.1 Determine the processes during deployment

Describe all processes that are running during deployment: Build, deploy and post-deploy phases. Explain why downtime occurs on your project

What role every process/phase plays and how to impact every process

How Magento Cloud deploys Magento. What every script does on every deployment phase

How to extend these scripts and best practices for doing so

Describe the ways to retrieve logs for phases and its scripts

5.2 Demonstrate the ability to create Magento Cloud script configurations