Thursday, June 30, 2016

ElasticSearch Client with Request Signing for AWS Recap

If you are trying to use AWS ElasticSearch and want to keep it secure you will hit the ‘brick-wall’ of frustration pretty quickly.  AWS ElasticSearch can be secured via 

  1. IP
  2. IAM Resource or Role


However, getting things to work with common out of the box mechanisms is not there. 

Problem for securing via IP
If you are running your client from a single server with a fixed IP, then it is easy.  If you are running from spot instances or lambdas then control of your IP is complicated.   Plus changing the security policy on AWS Elastic Search is minutes before it is in effect.

Problem for securing via IAM or Role
Most of us use the standard ElasticSearch client in our language of choice.  By default, this client talks direct HTTP with some built-in authentication mechanisms.  The client does not support AWS request signing, however, it does have a pluggable mechanism. 

Solutions for IP Mechanism
Elad Nava wrote the go to post on getting it done with a nginx proxy behind a VPC.
https://eladnava.com/secure-aws-elasticsearch-service-behind-vpc/

Now your connection flow
client(s) --> PROXY in VPC --> AWS ElasticSearch
can be secured via the proxy and standard AWS security groups.  While your Proxy and AWS ElasticSearch are hardened to a single IP.   

Solution for using IAM or Roles
While the default client does not support signing requests, it does allow a handler to modify the request.  If someone develops a handler for each language specific client then you can access AWS ElasticSearch from the standard client library!   However, I could not find a reference or link to these handlers on AWS or ElasticSearch docs, instead scattered around mentions in forums.  I pulled together the ones I required and linked them below. 

Special thanks to the folks that got the ball rolling on these clients and opened them up.

PHP

Python

NodeJS




Posted by at No comments:

Monday, June 27, 2016

New Jersey Teacher of Students with Disabilities Research

This research was for someone (who shall go nameless) that wants to learn more about extending their teaching knowledge and capabilities.  Specifically to gain a "Teacher of Students with Disabilities" endorsement/certificate.

Last Updated: 6/27/2016

General Notes: 
Rowan : Of all the sites - Only Rowan is clear about price and online availability.  Pricing from all other schools is either hidden or broken into multiple fees.
Cost:   I tried to use the higher cost of the per credit to reflect cost.  Though beware each school has a host of ‘fees’. All schools have some form of one-time application fees ranging from 50 to 100$.   I used in-state and part-time pricing for all selections. 

SchoolProgramCreditsCost/CreditCost Link
Rowan Link21730/crCost Link
College of New Jersey Link24876/crCost Link
Rutgers Link21900/crCost Link
Centenary Link21881/crCost Link
Kean Link21772/crCost Link
Rider Link21760/crCost Link
William Patterson Link22688/crCost Link
Seton Hall University Link211171/crCost Link
Richard Stockton College of New Jersey Link21757/crCost Link
Bloomfield College Link21620/cr(??)Cost Link
Ramapo College Link21726/crCost Link
Fairleigh Dickinson University Link21719/crCost Link
Caldwell College Link21900/crCost Link
Felician College Link211000/crCost Link
College of Saint Elizabeth Link211001/crCost Link
Saint Peter's College Link211050/crCost Link
Georgian Court University Link :pg 8821839/crCost Link
Montclair State University Link24655/crCost Link
Monmouth University Link211273/crCost Link
New Jersey City University Link21660/crCost Link

* Cost is per credit. Evaluated as part-time, in-state, if mentioned with fees.  See below for details or click Cost Link to see source. Note: many schools have other fees, please read source.

Rowan


College of New Jersey

Rutgers


Centenary 


Kean


Rider


William Patterson


Seton Hall University


Richard Stockton College of New Jersey


Bloomfield College


Ramapo College


Fairleigh Dickinson University


Caldwell College


Felician College - Rutherford Campus


College of Saint Elizabeth


Saint Peter's College


Georgian Court University


Montclair State University


Monmouth University


New Jersey City University


Posted by at No comments:

Monday, May 9, 2016

Lambda Environment Configuration with DynamoDB and /tmp


Currently AWS does not have a mechanism for Lambda deployment to encapsulate environment specific configuration.  You must include configuration files alongside your Lambda functional code.
When working on a photo architecture (https://runsignup.wordpress.com/2016/03/29/runsignup-photo-architecture/) we require ability to be a bit more dynamic in our approach for configuration and not depend on code updates.  We decided on using DynamoDB for managing this configuration, and improving performance by taking advantage of the /tmp storage available in container re-use as described here.   Below are the options we reviewed and the code for using Lambda + Dynamo + /tmp storage.

Here are some options for being more dynamic and code example for the approach we used.

[Note: If you are using API Gateway there is a better workaround, read here, we require S3 triggers]

a.  Update Lambda every time configuration changes and redeploy. 

Having configuration inside Lambda is a good from a performance perspective, but managing many customer specific Lambda's would be problematic.

b.  Use the Lambda description to encapsulate config object.

A hack that works (confirmed) and detailed here, however there is a  256 character limit for description field per documentation.

c. Use S3 to store the configuration file.

This allows remote storage of configuration via a file.  What's better Dynamo as document or S3 as a file?  Not too much, though the folks at Concurrency Labs had the same thought process and did a some performance measurements.  You can read more about their insights at their blog post here, the following is from their results.

d. Use a more dynamic approach with DynamoDB

Our approach was to use DynamoDB for the configuration store, while using /tmp to gain performance benefit when the container is re-used. 



Posted by at 1 comment:

Monday, January 25, 2016

AWS Certificate Manager, Easy Button


I know the pains of managing SSL certificates for a data center or even the cloud.  It is a manageable pain, you can live with it, but no one wanted to deal with the pain.  

  • Acquiring an SSL Cert was never automated
  • Deploying SSL Cert to network gear had challenges
    • Automating between different vendors
    • Keeping scripts updated
    • Adding/Removing network gear
    • Documenting the process, so next person could do it
  • Calendar/Scripts to notify when CERTs are expiring
  • Project Manager reminding people to do the work
I recently built a RunSignUp racing results app using Ionic/PhoneGap/Angular which is built for Web, Android, and iOS.  One of the items on the to-do list - setup SSL cert.  I was dreading this task, the details of getting the moving parts correct is a time sink.  Well here is a case where slight procrastination worked out pretty well - AWS announced the AWS certificate manager. 


1. Add your domain name


2. Review your request


3. Wait for validation

We can see in status in list view as well





4. Site owner receives and responds to email


5. We are now ready to go!


6. Before the CERT was deployed


7. We can now make our cloudfront domain use the cert.


8. Send a Cache Invalidation

9. And now we are green and serving on SSL


10. Change CloudFront behavior to alway redirect to HTTPS






Posted by at No comments:

Wednesday, January 13, 2016

Gulp bump minor number for ionic

If you need a full blown approach for bumping your version numbers then there is a robust set of build tools by Robert Shedd 
https://github.com/shedd/ionic_build_tools#bump-app-versions
His build tools provide more than just gulp commands, the readme alone can save you time and help you avoid mistakes.

However, I just needed something simple for bumping version numbers in package.json and config.xml.

Extra NPM packages

  • npm install gulp-xml-editor --save-dev
  • npm install gulp-json-transform --save-dev 


Requires

  • var jsonTransform = require('gulp-json-transform');
  • var xeditor = require('gulp-xml-editor');

Gulp Task
gulp.task( 'package', ['sass'], function(){
gulp.src('./package.json')
.pipe( jsonTransform(function(data){
var version = data.version.split('.');
version[2]++;
data.version = version.join('.');
return data;
},2))
.pipe(gulp.dest('.'));
gulp.src("./config.xml")
.pipe(xeditor(function(xml, xmljs) {
var version = xml.root().attr('version').value().split('.');
version[2]++;
  xml.root().attr('version',version.join('.'));
  return xml;
}))
 .pipe(gulp.dest("."));
});





Posted by at No comments:

Thursday, January 7, 2016

Bucket List item Done - Firefox Plugin

It may have taken a decade and a good reason but I finally wrote a Firefox Plugin!

Why Firefox and not Chrome?  

It is an integration with Selenium IDE which is a Firefox extension.

What does it do?

Provided integration with Selenium IDE to record browser traffic and save results as JMeter load test (.jmx file).  You can turn it on, run your Selenium test, turn if off and then you have JMeter version of your Selenium test.  Take the output and load it directly into JMeter or run it scale on RedLine13.

Why did I write this?

A use case I hear often is how to scale real user testing.  The typical answer is run some Selenium tests in parallel with JMeter tests.  There was no quick way to play your selenium test and record a JMeter test.  Problem solved :)

Did you enjoy writing this plugin?

Yes and No.  Yes, had to learn something new.  Once you learn how to integrate with Firefox (or any system) so many other possibilities open up.
No - I had to use the XUL system instead of the update Firefox extension mechanism.

Where is the code?

I made it GPL and opened it up https://github.com/redline13/selenium-jmeter.
Any ideas for extending or contributing would be welcome.

How do I use it?

I wrote the intro and tutorial at RedLine13 blog

  • https://www.redline13.com/blog/2016/01/introducing-selenium-ide-jmeter-plugin
  • https://www.redline13.com/selenium-jmeter-plugin-tutorial

Questions?

If you have any questions feel free to send me a note, but other public mechanisms would be

  • Pull request if you want to add something
  • Ask a question in RedLine13 forum  https://groups.google.com/forum/#!forum/redline13
  • Get a basic RedLine13 subscription and will provide support and get feature requests into the pipeline
Posted by at No comments:
Subscribe to: Posts (Atom)