promet/solr-mc repository overview

⁠Martin County

Build Status (develop):

NOTE: Running this project on the Mac or Windows requires that you have Docker for Mac Community Edition (edge) installed. If you are running the stable version you will encounter performance issues that will make this project extremely slow, and you will encounter gateway errors in certain instances.

⁠Prerequisites (2017-06-20)

In order to stand up a development environment using Docker, you need to have the following:

• Make (Mac Users you should have this)
• Docker and Docker Compose installed (or Docker for Mac / Windows).
• A Public key installed on the Promet Martin County dev / staging server
• A Public key installed on Acquia Cloud, with Team Lead access to the Martin County project (this is temporary, and will be changing in the near future.)

⁠Docker Stack Installation Instructions (2017-06-20)

The install process is multistep:

1. Create a ssh alias (see below)
2. Clone this repository into a directory called "martincounty"
3. Sync the files directory from Acquia Cloud, non-document files only (no .txt, .pdf, or .PDF files synced) (make sync-files-directory)
4. Copy the file docker/docker-github.yml to the project root, and add your Github Authentication token (see below for more info)
5. Add "local.martincounty.us" to your /etc/hosts file, pointed at IP Address 127.0.0.1
6. Build your local development environment using the Makefile (make initialize)

When you've completed the above steps, your site will be available at http://local.martincounty.us:8080⁠

To shut down your development environment use the command make down. This will destroy your containers, shut down Docker sync and remove any data volumes. These will get recreated when you execute the command make initialize.

⁠SSH Alias (2017-06-20)

The Makefile has syncing commands that assume that you have a SSH alias set up on your local dev computer. That alias needs to be named "martincodev".

In order to set up a SSH alias, you will need to have your public key installed on the martincounty.prometdev.com server, and you need to know your username.

You also need to know the location on your development machine of your private key. Unless you are storing them in another place, they are typically located in the folder ~/.ssh. By default your private key is named id_rsa, although you can use another name for your key.

To add a SSH alias, edit (or add) the file ~/.ssh/config and add the following:

host martincodev
     User <username>
     HostName martincounty.prometdev.com
     IdentityFile <path/to/privatekey/id_rsa>

Save this file, and attempt to ssh into the server with the command "ssh martincodev". If you are successful, everything is set. If you get an error message, please file an issue with the SysAdmin team regarding troubleshooting your server access.

⁠Github Authentication Token (2017-06-20)

Because of GitHub's rate limits on their API it can happen that Composer prompts for authentication asking your username and password so it can go ahead with its work. This will break the automated build process using Composer inside a Docker container.

You can generate an authentication token for Github here: https://github.com/settings/tokens⁠

If you already have an authentication token for Github stored locally for Composer, you can use that token. For Mac / Linux users that token is found in ~/.composer/auth.json

Copy the file docker/docker-github.yml into the project root where the docker-compose.yml file is located, and edit the file to add your token to the GITHUB_AUTH_TOKEN environment variable under the php service key.

⁠Database (2017-06-20)

The database file is available on the Promet Martin County dev / staging server, and will be pulled automatically if a copy is not available locally in build/ref. This database is updated nightly; if you need a new copy you can obtain one using make refresh-db.

⁠Files directory Sync (2017-06-20)

Run make sync-files-directory to sync the non-document files from Acquia Cloud (this syncs from the dev server). This will get you the aggregated CSS / JS files, and all of the images on the site, but will not pull the document files.

⁠Building the search index (2017-06-20)

If you're working on any code related to search, you can build the search indices on your local machine with the following command:

make reindex-solr-search

This may take a while to build. You may have to run make index-solr-search if the above command times out. Typically it takes three runs to index all nodes (~35,000 items indexed).

⁠Makefile (2017-06-20)

To see all of the targets in the Makefile, you can issue the following command:

make help

⁠Cron Jobs (linked document needs to be updated)

There are a number of cron jobs on production in Acquia Cloud that keep the site running properly. Most of these jobs are involved with pulling document metadata from Documentum and queueing those up for processing by Tika, and subsequent indexing by Solr, and for processing the queue.

These jobs are currently documented in [[CRONJOBS.md]]. If these jobs are changed, [[CRONJOBS.md]] needs to be updated accordingly. It is the only documentation that exists for these, as they are not under version control.

⁠Developer Notes

⁠Drupal Core

The following patches have been applied to Drupal Core:

https://www.drupal.org/node/1697570#comment-11901028⁠ https://www.drupal.org/node/1311820#comment-9581751⁠ https://www.drupal.org/node/2821441#comment-11764648⁠

Note the following patches, not applied here but applicable:

This patch is a replacement for https://www.drupal.org/node/1697570#comment-11901028⁠: https://www.drupal.org/node/1978176#comment-11094363⁠

This patch is a reroll of https://www.drupal.org/node/2821441#comment-11764648⁠ https://www.drupal.org/node/1697570#comment-11901028⁠

⁠Modules of note

The date in parentheses after each module is the date the notes were last updated in this README. If you add or change the notes, please update the dates accordingly.

⁠Audit Log (17 June 2017)

This module was previously patched with patch audit_log-notice_in_auditlog_construct-2596653-2.patch.

This patch was reviewed and merged into the dev branch on 29 Jan 2016, along with some other patches that addressed other issues. After a review of the differences between the dev version and the stable version, a decision was made to eliminate patching the stable version in favor of installing the dev version of this module.

The Dev version installed is the version available as of 2016-Feb-04.

⁠Match Redirect (17 June 2017)

This module has been patched to address an issue identified in https://www.drupal.org/node/2094217⁠. The patch submitted does partially address the problem. The patch name does not comply with drupal coding standards, so the issue number is noted here. Note: this patch creates an undocumented dependency on the contrib module Entity API.

To fully address this issue if it recurs, you may also have to clear the Memcached cache. In the rare event that this needs to be done on Acquia Cloud, instructions can be found here:

https://docs.acquia.com/article/memcache-monitoring-and-flush-using-nc-command⁠

⁠Add To Cal (20 Jun 2017)

This module is a patched version of 7.x-1.0.

The following patches have been applied:

https://www.drupal.org/node/2470347#comment-12136575⁠ https://www.drupal.org/node/2578763#comment-10400609⁠

⁠AdvancedQueue (20 Jun 2017)

This module is a patched version of 7.x-1.0-beta3.

The following patches have been applied:

https://www.drupal.org/node/2854318#comment-11947374⁠

⁠Date Repeat Entity (20 Jun 2017)

This module is a patched version of 7.x-2.0.

The following patches have been applied:

https://www.drupal.org/node/2447949#comment-9695939⁠

Note: This should be revisited, there is a more recent patch that is more relevant, but currently needs to be rerolled.

⁠Linkchecker (20 Jun 2017)

This module is a patched version of 7.x-1.2.

The following patches have been applied:

https://www.drupal.org/node/2609460#comment-10534040⁠ https://www.drupal.org/node/2610188#comment-10537832⁠

*Note: The second patch, 2610188, was created by A. Chappel, but is disputed by the module maintainer. See the discussin thread for more information.

This patch creates a dependency in Linkchecker on the AdvancedQueue module.

⁠Responsive Image (20 Jun 2017)

This module is a patched version of 2.0-dev, pinned at commit #5176491

The following patches have been applied:

https://www.drupal.org/node/1706596#comment-10502560⁠

⁠Search API (20 Jun 2017)

This module is a patched version of 1.0-dev, pinned at commit #5990977

The following patches ave been applied:

https://www.drupal.org/node/2505421#comment-10019375⁠

Note: There is a dispute about this patch from the module maintainer on the discussion thread.

⁠Token Variable (20 Jun 2017)

This module is a patched version of 7.x-1.3.

The following patches have been applied:

https://www.drupal.org/node/1879804#comment-7316380⁠

⁠User Alert (20 Jun 2017)

This module is a patched version of 7.x-1.10.

The following patches have been applied:

https://www.drupal.org/node/2726829#comment-11974812⁠

⁠Views Load More (20 Jun 2017)

This module is a patched version of 7.x-1.5.

The following patches have been applied:

https://www.drupal.org/node/1475642#comment-9113437⁠

Note: there are more recent versions of this patch that should be investigated and potentially used in lieu of the one documented here.

⁠martin_county_content_workflow features module (20 Jun 2017)

Whenever we removing a role, We make sure that we search for any occurrences of that role that may be left in code after we delete the role from the “roles” page in the Admin UI (grep is a good tool for this

grep “<user role>” -n -r *

You can run that inside the modules folder and it will return the filename, and line number where the string in quotation marks occurs). If there are any occurrences you will need to delete them manually from code.

The permissions will be housed in code not only in the martin_county_content_workflow.features.user_permission.inc but also in the martin_county_content_workflow.info file; if you are removing a workbench moderation permission completely you may have to manually delete it from the .info file, as features doesn’t always remove those cleanly.

A clue that there is problem will occur when you build your local environment from scratch, and you start seeing multiple occurrences of this error when you revert features:

in_array() expects parameter 2 to be array, null given features.user.inc:151

Moderation state option, 'Needs Review' is currently disabled via, hook_workbench_moderation_states_next_alter(). Please see, martincounty_helper_workbench_moderation_states_next_alter() on martincounty_helper.module.

⁠Development Tips (20 Jun 2017)

To get information on development tips, deployment, and information about your local development machine, continue reading.

This project is also hosted on Acquia⁠.

• Promet Dev⁠
• Acquia Dev⁠ machine name of "dev" for drush and jenkins
• Staging⁠ machine name of "test" for drush and jenkins
• Prod⁠

This project has automated testing set up for Pull Requests and automatic pushes to both Promet Dev and Acquia Dev.

This project uses the idea of an artifiact repository located on Acquia at [email protected]⁠:martin.git Please see the Deployment To Acquia⁠ section below.

⁠Compass Compile

Line comments, or the sass debug info, is removed when we do compass compile. However, for local development, you may opt to see sass debug info in the compiled css for debugging purposes. To do this, just set the environment to development i.e., compass compile --environment=development.

⁠Deployment to Acquia (20 June 2017)

⁠With Jenkins

1. Navigate to Martin Co Dev Deploy to Acquia⁠
2. Click on "Build with Parameters"
3. Replace $GIT_COMMIT with the ID of the commit you wish to push. Note: This is easiest found on github or in git log Ex: e359b2b01242e71e5b384774d278241f7b1615b2
4. Choose the Branch you want to push to. Usually the same as the one you are pulling from. Ex: release-2.0.0.
5. Choose the environment you wish this to deploy to. "dev" and "test" are your options.
6. Click Build

After pressing build, Jenkins will push a rendered version of the site (runs composer, puts configuration where it needs to be, etc.) to the acquia repository to the branch you declared. It then changes the deployed code of the environment you specified, then fires the "Martin Co Acquia Sync Database" task on the environment of your choice. This runs the cloud hooks needed to run update.sh⁠

⁠Note: This readme is a work in progress. Information past this line may be dated or inaccurate.

⁠Pushing to Production (needs review)

Due to the cloud hooks that are in place, pushing to production is a breeze.

1. Checkout the acquia repository somewhere you can access it.
2. Checkout the branch you are releasing. Note: the Acquia Repo is an artifact repo. That means it doesn't follow the rules of gitflow. So if you pushed to staging using release-1.2.3, then you will have to checkout release-1.2.3 to have the commit that was pushed to it.
3. Create a tag on the latest commit in that branch.
4. Push that tag to the Acquia Repository
5. Go to Acquia's Workflow Control Panel for Martin County.
6. Back up the Database
7. Deploy the Tag you just created.
8. Validate there are no errors in the log for code deployment. NOTE: the log there will show you the output of the script. Just click the "Details" link next to the task that says "Deploy code to prod".
9. Clear Varnish Cache.
10. Verfy the site loads.

⁠Custom Drush Commands and Special Setup (needs review)

This project does have a couple of special setup items that need to be taken in account for local development.

⁠Documentum Queue

To test documentum, this has to pull from the Martin documentum server. To accomplish this, we have set up a drush command that builds queue items. The queue is implemented using AdvancedQueue⁠. On prod these steps are implemented in a series of cron jobs.

The gist of the workflow is like this:

Custom Drush Command drush mc-pull-files

• Pull down the document records from the Documentum API.
• Check to see if the record has changed.
• Check to see if the Document's Modified Date has Changed
• If the Record doesn't exist or the Modified Date has changed, send the record to the appropriate queue.

In the queue drush advancedqueue DOCUMENTUM_QUEUE_NAME

• Send the PDF to a Promet maintained PDF Parser⁠ to grab the text.
• Create a node with the text and a bunch of the meta data from the API Record

To use locally, simply do the following:

• drush mc-pull-files [--types]
  • --types is a comma separated list of types defined in the code. Currently, this can be any of bid, agenda, agenda_item, corespondence, info_release. NOTE: yes, corespondence is mispelled on purpose.
• drush advancedqueue --all NOTE: advancedqueue will run forever if you let it. You will have to kill it with CTRL-C. Use drush advancedqueue-list to see if your queue is done running.

If you need to do PDF Parser development, you can easily do so on this vagrant machine. PDF Parser is pulled down to your machine and is running on port 8081. You can see it in build/vagrant.sh where it is started for you on provision. An example request can be.

http://martincounty.dev:8081/http%3A%2F%2Fwww.pdf995.com%2Fsamples%2Fpdf.pdf

⁠SimpleSAML (needs review)

This project uses a service provider that allows Single Sign On. This is called SimpleSAMLphp⁠. This is NOT Drupal specific and is the only Acquia supported SSO method.

You will not be able to use this locally, You will need to test it on one of the following servers.

Acquia Prod: 107.21.97.85 54.243.131.134 54.243.132.39

Acquia Stage and Dev 54.225.165.124

MartinCounty.PrometDev.com 104.130.70.101

⁠solr (needs updating)

Your machine has a copy of Solr installed on it. Your Search API server connection will be automatically generated to connect to Solr. Use built in drush commands to interact with Search API.

Keep in mind the database has a now outdated references between YOUR solr and YOUR drupal, because it was referencing PRODUCTION'S solr before. If you are doing anything that involves testing search... in order to get the correct results YOU MUST REINDEX

To reindex search in Drush run:

drush search-api-clear
drush search-api-index

This is the case on Acquia Staging and Acquia Dev as well. You will find a Jenkins Job⁠ that will allow you to reindex either dev or staging. Simply input the environment as dev or test. This WILL work on production as well but only use under EXTREME circumstances.

⁠XML Sitemap (needs review)

XML Sitemap will be working at the production site in https://www.martin.fl.us/sitemap.xml⁠. Other environments, namely the prometdev, acquia dev, and acquia stage may return a blank sitemap.xml page. The acquia prod https://martin.prod.acquia-sites.com/sitemap.xml⁠ will also return a blank sitemap.xml page, even though it is also martin.fl.us. This is due to the fact that xml sitemap remembers the settings it was last saved. For example, using firefox, if you visit https://martin.prod.acquia-sites.com/sitemap.xml⁠, you will see the following error:

Error loading stylesheet: An unknown error has occurred (805303f4)
https://www.martin.fl.us/sitemap.xsl

If we save settings using the acquia prod, the martin.fl.us site will be the one showing the blank xml page and the above error trying to load stylesheet from the other domain. On that note, all settings for xml sitemap shall be done using the main domain.

⁠docker-sync

Docker sync has helped performance locally.

To implement that (on Mac OS X):

brew install unison
brew install eugenmayer/dockersync/unox (if error installing, may have to "brew install python@2")
gem install docker-sync
docker-compose -f docker-compose.yml -f docker-compose-mac.yml up -d
docker-sync start