Karpenter vs ECK Elasticsearch

Posted on August 27, 2025 by

AWS Karpenter vs ECK Elasticsearch

Fixing ECK Elasticsearch Cluster Issues with Karpenter and PodDisruptionBudgets

Recently, we ran into an issue with our Kubernetes cluster where Karpenter was updating AMIs. While this normally helps keep nodes up-to-date, in our case it was replacing nodes faster than Elasticsearch could recover. The result was a large Elasticsearch cluster that ended up in a state where all pods were stuck in Pending.

This happened because the cluster could not maintain enough master pods online at the same time to form a quorum. Without quorum, Elasticsearch cannot elect a leader or coordinate writes, which effectively stalls the entire system.

Initial Symptoms

When this problem hit, the cluster looked deceptively healthy at first glance. All the nodes were technically “running,” but the Elasticsearch pods never reached a true Running state. Looking deeper into the pod logs revealed repeated messages that the nodes could not discover enough masters to join the cluster.

Our Attempted Fixes

Restarting pods one by one didn’t work. In fact, it often made the problem worse, since the cluster still couldn’t reach quorum with staggered pod restarts. Elasticsearch remained stuck, waiting for enough master nodes to reappear.

The eventual solution was drastic but effective: 

kubectl -n <namespace> delete po --all

This forced all Elasticsearch pods in the namespace to restart at the same time. By doing so, Elasticsearch was able to reinitialize its bootstrap discovery process, elect new masters, and reform the cluster correctly. Thankfully, no data was lost during this reset.

Why We’re Cautious

We’re always hesitant to do full cluster restarts like this. In the past, we’ve seen situations where cached data issues forced us to delete underlying PVCs (persistent volume claims), which is a last-resort scenario because it carries the risk of data loss. In this case, we were lucky—the cluster healed itself without having to delete PVCs—but this experience highlighted the need for better safeguards.

Preventing This in the Future

The real lesson here is that clusters running stateful applications like Elasticsearch need properly configured PodDisruptionBudgets (PDBs). A PDB ensures that Kubernetes (or in our case, Karpenter) doesn’t evict or replace too many pods at once, which would prevent Elasticsearch from maintaining quorum.

Here’s an example PDB definition we now apply to our clusters: 

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: elasticsearch-pdb
  namespace: elasticsearch
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      elasticsearch.k8s.elastic.co/cluster-name: bgt-bps

With this in place, Karpenter (or any node autoscaler) will be forced to wait until an Elasticsearch pod has fully come back online before it can evict another. This ensures the cluster always has enough master nodes available to maintain quorum, significantly reducing the risk of a total cluster stall.

Conclusion

This incident was a good reminder that while Kubernetes and tools like Karpenter provide incredible automation, stateful workloads like Elasticsearch require extra guardrails. Without PodDisruptionBudgets, an automated upgrade cycle can accidentally take down an entire cluster.

The combination of careful monitoring, safe disruption policies, and a solid disaster-recovery plan ensures that Elasticsearch can run reliably—even in the face of automated infrastructure updates.

Posted in Elasticsearch, kubernetes | No Comments »

Solar Panels For EV Cars

Posted on August 27, 2025 by

Solar Car Dream

When I was a kid, I often dreamed of having a car that was fully electric and could run on solar power in real time. Imagine never needing to stop to recharge—just endless travel powered by the sun. A vehicle like this would allow you to roam the world without ever worrying about fuel stations, gas prices, or plugging in overnight.

Fast forward to today: nearly every major car company is producing electric vehicles (EVs), largely thanks to Tesla’s success in proving the market. EVs are now more popular and financially accessible than ever before. That naturally leads to the question:
Can we put solar panels on EVs to drive them in real time using only sunlight?

Feasibility

To figure this out, we’ll look at a real-world example using Tesla data and standard solar cell efficiency numbers.

Tesla Real-Time Power Requirement

According to Tesla energy usage statistics, a Tesla consumes about 34 kilowatt-hours of electricity to drive 100 miles. That’s roughly the energy requirement we need to meet if we want to run the car on sunlight alone.

Surface Area of a Tesla Vehicle

Let’s estimate the surface area of a Tesla sedan that could reasonably be covered with solar panels. For simplicity, we’ll just use the roof and upper body where direct sunlight hits, and balance out glass/window areas with some side paneling.


Tesla vehicle dimensions

Using Tesla’s published vehicle dimensions:

  • Length: 196 in
  • Width: 77.3 in (excluding mirrors)

Surface area (top view) = 77.3 in × 196 in ÷ 144 (to convert to square feet).
That works out to about 105.3 square feet of usable solar surface.

Solar Power Generated Per Square Foot

Modern solar cells are around 15% efficient and typically generate about
15 watts per square foot in direct sunlight.

That means our 105 sq. ft. of solar coverage could generate approximately:
105 × 15 = 1,575 watts (1.575 kW) of power per hour.

Putting the Pieces Together

Here’s the problem: a Tesla requires about 34 kWh to travel 100 miles.
But our entire car surface covered in solar panels only provides 1.575 kW per hour.

That’s just 4.6% of the power needed to drive in real time under perfect sunlight conditions. In other words, you’d still need an additional 32.5 kW to make the car fully solar-powered.

To generate that with current solar technology, you’d need a trailer covered in solar panels with a surface area of about 2,166 square feet—roughly the size of a small house!

Clearly, while the dream is exciting, the math shows that it’s not currently realistic.

Future Possibilities

Does that mean solar cars will never happen? Not necessarily. As solar efficiency improves—moving toward 30–40% or even higher—vehicles may someday harvest enough energy directly from their panels to meaningfully extend driving range.

In fact, companies like Aptera and Lightyear are already experimenting with ultra-efficient EVs that integrate solar panels into their designs. While they don’t yet achieve “infinite driving,” they do gain dozens of free miles per day from sunlight alone.

So for now, my childhood dream of a truly solar-powered car remains out of reach. But given the pace of technology, it may only be a matter of time before driving on sunshine becomes more than just a dream.

Posted in Solar Power | No Comments »
Tagged: , , , ,

ECK – Unify Kibana and Elasticsearch Versions

Posted on August 26, 2025 by

Managing Elasticsearch and Kibana Versions with ECK, ArgoCD, and Kustomize

We use the ECK operator to manage our Elasticsearch clusters. It does a great job and integrates well with our GitOps workflow.

All of our deployments run through ArgoCD, and we use Kustomize to tie the pieces together. This works smoothly in most cases, but there is one recurring problem.

The Problem

When updating Elasticsearch, we sometimes forget to update Kibana. The issue is that Kibana is defined in a separate YAML, and with so many clusters it’s easy for the versions to fall out of sync.

Why This Is a Problem

Elasticsearch and Kibana versions need to be compatible. If Elasticsearch is upgraded but Kibana isn’t, things break. With dozens of clusters, missing an update becomes a very real risk.

What we need is a form of inheritance—a single place to define the version that applies to both Elasticsearch and Kibana.

The Solution

The solution is to use the replacements function in the Kustomization resource. This ensures that the version defined in the Elasticsearch resource is automatically propagated to Kibana.

kind: Kustomization
apiVersion: kustomize.config.k8s.io/v1beta1
replacements:
  - source:
      kind: Elasticsearch
      name: bgt-bps
      fieldPath: spec.version
    targets:
      - select:
          kind: Kibana
        fieldPaths:
          - spec.version

Now, the Elasticsearch YAML becomes the authoritative source for the version. Whenever ArgoCD syncs, the spec.version value is copied from Elasticsearch to Kibana automatically. This prevents drift and ensures the two always stay aligned.

Posted in kubernetes | No Comments »

AI Longhorn Destruction + Solution

Posted on August 22, 2025 by

Longhorn PVC Mounting Issue and Backup Setup

Recently I ran into a persistent error with Longhorn while trying to attach a PVC. The error kept repeating and initially, I thought I needed the v2 engine—but it turns out the real culprit was multipath being created. The
Longhorn KB article on troubleshooting multipath explained exactly how to fix it.

Original Error

This was the original error I kept seeing:

MountVolume.MountDevice failed for volume "pvc-4b04c380-e38a-4462-a71a-875a16b53c9d":
rpc error: code = Internal desc = format of disk "/dev/longhorn/pvc-4b04c380-e38a-4462-a71a-875a16b53c9d" failed:
type:("ext4") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/.../globalmount")
options:("defaults") errcode:(exit status 1)
output: mke2fs 1.47.0 reports disk is in use; will not make a filesystem here!

I tried several fixes, but none worked. Every time I would try and restart all longhorn pods like iAI recommended the errors became like a game of wacamole. Randomly other pvcs would have the same issue. Eventually, AI Claude 4 led me down a path that deleted all the Longhorn drives, and I had no backups—so this was a costly lesson.

Final Solution

The final solution was to go on each node and set the multipath setting as explained in the article. I also realized the importance of having a backup solution in place.

MariaDB Backups to S3

For some databases, I have a MariaDB backup YAML running that sends all objects to S3. Here’s an example (sensitive data sanitized):

apiVersion: k8s.mariadb.com/v1alpha1
kind: Backup
metadata:
  name: mariadb-backup
spec:
  mariaDbRef:
    name: mariadb
  storage:
    s3:
      bucket: my-bucket
      prefix: mariadb-backups/
      endpoint: s3.amazonaws.com
      region: us-east-1
      tls:
        enabled: true
      accessKeyIdSecretKeyRef:
        name: aws-credentials
        key: ACCESS_KEY_ID
      secretAccessKeySecretKeyRef:
        name: aws-credentials
        key: SECRET_ACCESS_KEY
  schedule:
    cron: "0 2 * * *"
  maxRetention: "720h"  # 30 days
  compression: gzip
  databases:
    - mariadb
    - recisphere

Longhorn Backup Setup

For Longhorn, I first updated the values.yaml with the backup settings (sensitive data sanitized):

defaultSettings:
  backupTarget: "s3://user@us-east-1/longhorn-backups"
  backupTargetCredentialSecret: "s3-backup-secret"
  allowRecurringJobWhileVolumeDetached: ~
  createDefaultDiskLabeledNodes: ~
  defaultDataPath: /mnt/sda

Next, I created a recurring backup job:

apiVersion: longhorn.io/v1beta2
kind: RecurringJob
metadata:
  name: daily-backup
  namespace: longhorn-system
spec:
  cron: "0 2 * * *"
  task: backup
  groups: ["daily-backup"]
  retain: 7
  concurrency: 2

Finally, I annotated all PVCs with the recurring job label:

kubectl get pvc --all-namespaces --no-headers | while read ns name rest; do kubectl label pvc $name -n $ns recurring-job-group.longhorn.io/daily-backup=enabled --overwrite; done

metadata:
  labels:
    recurring-job-group.longhorn.io/daily-backup: enabled

Verification

Now the Longhorn UI shows a backup target connected to S3, and backups appear there. The cron job handles automatic backups, and you can view the job pod outputs in Kubernetes for debugging if necessary.

Lessons Learned

  • Always verify multipath settings on nodes to avoid PVC mount issues.
  • Never rely solely on local storage—set up automated backups for both databases and Longhorn volumes.
  • Test backup restores periodically to ensure the data is recoverable.
  • Use short, descriptive PVC names and recurring job labels to avoid confusion in large clusters.

Posted in AI, Code Solutions, Uncategorized | No Comments »

Enabling Windows nodes inside an EKS cluster on AWS.

Posted on September 27, 2023 by

Description

We recently went about adding windows nodes for some legacy dotnet stacks at Ancestry. As part of this we followed the AWS documentation to enable windows. We also use Karpenter to handle the scheduling and decided to make a new provisioner for just windows. After following the docs a new windows node came up but there was an error that prevented the pods from obtaining an ip address.

Warning FailedCreatePodSandBox 3m21s (x4555 over 19h) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a760b4d93ed0937341cb5083547b0b8a197a280a66ad3d0cb096562ab2a237a1": plugin type="vpc-bridge" name="vpc" failed (add): failed to parse Kubernetes args: failed to get pod IP address windows-test-5bbc88b5f9-vzgfc: error executing k8s connector: error executing connector binary: exit status 1 with execution error: pod windows-test-5bbc88b5f9-vzgfc does not have label vpc.amazonaws.com/PrivateIPv4Address

Solution

My suspicion was there was something in the control plane that can’t be seen from kubectl. After talking with AWS support through several rounds there is an admission webhook in the control plane that is triggered by setting the nodeSelector. The docs say that needs to be set, but I thought it was just to set nodeaffinity and not the other field.

nodeSelector:
        kubernetes.io/os: windows
        kubernetes.io/arch: amd64

Once we added this to the pod spec section of the deployment yaml the pods were able to successfully be triggered by the mutating webhook to add the right annotations vpc.amazonaws.com/PrivateIPv4Address to the pod and it was able to get an ip address succesffully.

Posted in Amazon Web Services, kubernetes | No Comments »

Birth of a Liberal

Posted on August 29, 2021 by

Abstract

I just want to begin by expressing my gratitude for this country where I live, The United States of America. It really is a land of immense freedom and opportunity. I love the constitution and feel that it really is an inspired document. I consider myself somewhere in the middle of the political spectrum. I do not like extremist views in either direction, conservative or liberal. The following is just my personal viewpoints and opinions on how liberals and conservatives are the same people just separated by time. Liberal views on the political spectrum are just a natural evolution of a progression where people live.

How to become liberal

The Wild West

In 1803 the United States made one of the largest land purchases in history known as “The Louisiana Purchase.” This purchase was for what is now the Midwest states such as Iowa, South Dakota, Kansas, Oklahoma, and of course Louisiana. Several years after this purchase the rest of the west joined the union. California through Utah came from the Mexican Cession in 1848, Oregon and Washington joined in 1846, and Texas joined the union in 1845. The United States had a plethora of land that needed people to fill it, and thus the Homestead Act was introduced in 1862 by President Lincoln. This act allowed people to claim up to 160 acres that was currently unclaimed at the time for free. The only requirements were the owners had to live on the land, improve it, and pay the registration fees. The Homestead Act caused many people to move west with the dream of owning their own land for free.

Try and imagine living during this time and deciding to travel west to take advantage of the free land. You sell everything you have, which isn’t much by today’s standards. You buy a wagon that is pulled by either horses or oxen. You load the wagon with food, seeds for growing future food, tools, a gun with some ammunition, and maybe a few changes of clothes. Early in the spring you take your family and begin walking from one of the eastern states and traveling west. The average distance the pioneers would travel in one day was 15 miles. To cover the number of miles needed to head west this means you will be traveling for six months. There is an old game called The Oregon Trail that accurately describes life as a pioneer heading west.

After six months of traveling, you find a location that you can call home. There is a small town 15 miles away where you register your land. You have no house, there is now hospital, there are no neighbors. Just your small family and the local wildlife for company. There is much work to be done as winter is coming. You need to build a cabin or house to protect your family. You need to dig a well so you can pull fresh water. After winter comes and goes you need to get a harvest started, which means plowing the fields and preparing them to be farmed. There is a near endless amount of work to be done.

Living one 160 acres by yourself means you are the only rule and law around. With the culture at that time it was perfectly acceptable to shoot and kill any intruder or stranger that came to your property. Remember you are the only person living out there. You are the law. Whatever you say becomes the law. You have no neighbors to worry about. You in essence have the freedom to do anything you want.

The Hamlet or Township

Staying with this same scenario let’s advance forward a few years. You have had several children that are now married and want to live near you. Next to your house you divide off sections of land for each to build a house. Your children then have other relatives that visit and want to move close to them. More houses are built. Now you have a small township with ten to thirty homes. You are not free to do everything the same as you once did. The more people that come together and try to live together the more freedoms need to be surrendered for everyone to live peacefully together. You cannot just kill anyone that comes to your house. You cannot cause a disturbance to the peace in the middle of the night. Everyone in this small township must agree on certain rules to follow in order to have peace and they pick one person as a sheriff in order to maintain this law. To pay for the sheriff and various other publicly shared municipalities such as a road taxes now have to be levied. They aren’t huge and the people are only taxed when something needs to be done.

Even though you have given up a little bit of your ultimate freedom life is good because you no longer must wear so many hats. One of the women is a medicine woman and has a gift for healing, another resident is a baker, and each person finds unique tasks to specialize in so the whole community can share talent and resources.

Town, City, and Metropolis

As time progresses more and more people call this place home. There is an official town name given. The more people that come to live together the more municipalities there are that need to be created, such as schools, parks, public transportation, and etc. As any community grows in population there are more people that have to decide how to live peacefully together. This is no small task. Every step of growth more freedoms must be surrendered. Traffic lights for example inhibit people from speeding through intersections continuously, but they help give order so that everyone can share the road peacefully.

The difference in ideology between small communities and large cities can be seen today. Most of rural America has a fairly conservative base and the large metropolis cities have a more liberal view. Large cities such as New York City or San Francisco have different problems to solve than small towns like Andover South Dakota. Once upon a time San Francisco was a small farming community much like the small town in South Dakota, but as the city grew the problems that it faced changed and grew with it.

The next time you become upset becomes someone’s viewpoint is too liberal or too conservative this does not mean that either you or the other person is right or wrong. The difference is the time and growth of where you both live or grew up. Eventually the Earth will be just like Coruscant in Star Wars. Coruscant is a planet that is just one giant city. When this happens everyone will be relatively liberal because we will all have similar problems that we will be facing. At the end of the day we are more alike than we give ourselves credit. I believe that if a liberal person was forced to transplant into rural America that he or she would eventually become more conservative because the set of problems faced has changed. The same is true for a conservative person who moves into a metropolis megacity. Unfortunately, there are not many one solution fits all sizes for political issues. This is why I generally believe that states or local governments should be left to make decisions that are best for their residents.

 

Posted in Uncategorized | No Comments »
Tagged: , ,

Java New Relic Agent Increases Memory Footprint

Posted on July 27, 2021 by

Overview

We recently have been upgrading our java stacks to have the latest new relic agent 6 and 7 to incorporate distributed tracing into our environments. In testing the various versions of the new relic agent six we found that our stacks all took a performance hit. We increased our memory usage and increased our latency thus bringing us in danger of meeting our SLAs.

does new relic agent increase memory usage?

In this stack we did a roll back to a version 5 of the new relic java agent. The graph displays differently for the mysql database connection measurements. On the left is version 6.1 on on the right is version 5.3. There is a clear drop in memory usage when we use the older version of the java agent.

Version 7 Analysis

Right now we are currently testing various versions of the 7 release of new relic java agent. We are seeing an increased memory footprint still and many of our stacks are experiencing java garbage collection problems. This causes our stacks to periodically lock up and slow down response times thus triggering alerts. The issues will clean themselves up usually over a period of 30 minutes, however this creates way too many false alarms in our alerting process. Reaching out to the new relic support isn’t very helpful either. Just be cautious in any upgrades you do with the new relic agent after major version 5.

Posted in Uncategorized | No Comments »
Tagged: , , , ,

HSEARCH400080: Unable to detect the Elasticsearch version running on the cluster

Posted on July 21, 2021 by

Description

I recently decided to use the elasticsearch and hibernate bindings in java. Hibernate is already a very good ORM and has come a long ways in the last several years, but the integration with elasticsearch and lucene make it an amazing combination. It cuts down on the maintenance of keeping an elasticsearch index up to date and maintaining multiple classes for search providers. In my test project I decided to run the services using docker-compose. I ended up getting the following error that wasn’t very descriptive:

default] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.search.util.common.SearchException: HSEARCH000520: Hibernate Search encountered failures during bootstrap. Failures:

    default backend: 
        failures: 
          - HSEARCH400080: Unable to detect the Elasticsearch version running on the cluster: HSEARCH400007: Elasticsearch request failed: Connection refused
Request: GET  with parameters {}
Response: (no response)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1786) ~[spring-beans-5.3.6.jar:5.3.6]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:602) ~[spring-beans-5.3.6.jar:5.3.6]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524) ~[spring-beans-5.3.6.jar:5.3.6]
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.6.jar:5.3.6]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.6.jar:5.3.6]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.6.jar:5.3.6]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.6.jar:5.3.6]
    at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1154) ~[spring-context-5.3.6.jar:5.3.6]
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:908) ~[spring-context-5.3.6.jar:5.3.6]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.6.jar:5.3.6]
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:144) ~[spring-boot-2.4.5.jar:2.4.5]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:782) [spring-boot-2.4.5.jar:2.4.5]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:774) [spring-boot-2.4.5.jar:2.4.5]
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:439) [spring-boot-2.4.5.jar:2.4.5]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:339) [spring-boot-2.4.5.jar:2.4.5]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1340) [spring-boot-2.4.5.jar:2.4.5]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1329) [spring-boot-2.4.5.jar:2.4.5]
    at com.recisphere.ServerKt.main(Server.kt:54) [main/:na]
Caused by: javax.persistence.PersistenceException: [PersistenceUnit: default] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.search.util.common.SearchException: HSEARCH000520: Hibernate Search encountered failures during bootstrap. Failures:

    default backend: 
        failures: 
          - HSEARCH400080: Unable to detect the Elasticsearch version running on the cluster: HSEARCH400007: Elasticsearch request failed: Connection refused
Request: GET  with parameters {}
Response: (no response)
    at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:421) ~[spring-orm-5.3.6.jar:5.3.6]
    at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:396) ~[spring-orm-5.3.6.jar:5.3.6]
    at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.afterPropertiesSet(LocalContainerEntityManagerFactoryBean.java:341) ~[spring-orm-5.3.6.jar:5.3.6]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1845) ~[spring-beans-5.3.6.jar:5.3.6]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1782) ~[spring-beans-5.3.6.jar:5.3.6]
    ... 17 common frames omitted
Caused by: org.hibernate.search.util.common.SearchException: HSEARCH000520: Hibernate Search encountered failures during bootstrap. Failures:

    default backend: 
        failures: 
          - HSEARCH400080: Unable to detect the Elasticsearch version running on the cluster: HSEARCH400007: Elasticsearch request failed: Connection refused
Request: GET  with parameters {}
Response: (no response)
    at org.hibernate.search.engine.reporting.spi.RootFailureCollector.checkNoFailure(RootFailureCollector.java:50) ~[hibernate-search-engine-6.0.5.Final.jar:6.0.5.Final]
    at org.hibernate.search.engine.common.impl.SearchIntegrationBuilderImpl.prepareBuild(SearchIntegrationBuilderImpl.java:243) ~[hibernate-search-engine-6.0.5.Final.jar:6.0.5.Final]
    at org.hibernate.search.mapper.orm.bootstrap.impl.HibernateOrmIntegrationBooterImpl.doBootFirstPhase(HibernateOrmIntegrationBooterImpl.java:259) ~[hibernate-search-mapper-orm-6.0.5.Final.jar:6.0.5.Final]
    at org.hibernate.search.mapper.orm.bootstrap.spi.HibernateOrmIntegrationBooterBehavior.bootFirstPhase(HibernateOrmIntegrationBooterBehavior.java:17) ~[hibernate-search-mapper-orm-6.0.5.Final.jar:6.0.5.Final]
    at org.hibernate.search.mapper.orm.bootstrap.impl.HibernateOrmIntegrationBooterImpl.lambda$bootNow$7(HibernateOrmIntegrationBooterImpl.java:218) ~[hibernate-search-mapper-orm-6.0.5.Final.jar:6.0.5.Final]
    at java.util.Optional.orElseGet(Optional.java:267) ~[na:1.8.0_201]
    at org.hibernate.search.mapper.orm.bootstrap.impl.HibernateOrmIntegrationBooterImpl.bootNow(HibernateOrmIntegrationBooterImpl.java:218) ~[hibernate-search-mapper-orm-6.0.5.Final.jar:6.0.5.Final]
    at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) ~[na:1.8.0_201]
    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[na:1.8.0_201]
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_201]
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) ~[na:1.8.0_201]
    at org.hibernate.search.mapper.orm.bootstrap.impl.HibernateSearchSessionFactoryObserver.sessionFactoryCreated(HibernateSearchSessionFactoryObserver.java:41) ~[hibernate-search-mapper-orm-6.0.5.Final.jar:6.0.5.Final]
    at org.hibernate.internal.SessionFactoryObserverChain.sessionFactoryCreated(SessionFactoryObserverChain.java:35) ~[hibernate-core-5.4.32.Final.jar:5.4.32.Final]
    at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:385) ~[hibernate-core-5.4.32.Final.jar:5.4.32.Final]
    at org.hibernate.boot.internal.SessionFactoryBuilderImpl.build(SessionFactoryBuilderImpl.java:468) ~[hibernate-core-5.4.32.Final.jar:5.4.32.Final]
    at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:1259) ~[hibernate-core-5.4.32.Final.jar:5.4.32.Final]
    at org.springframework.orm.jpa.vendor.SpringHibernateJpaPersistenceProvider.createContainerEntityManagerFactory(SpringHibernateJpaPersistenceProvider.java:58) ~[spring-orm-5.3.6.jar:5.3.6]
    at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:365) ~[spring-orm-5.3.6.jar:5.3.6]
    at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:409) ~[spring-orm-5.3.6.jar:5.3.6]
    ... 21 common frames omitted


Process finished with exit code 1

Solution

After doing some digging I realized that the error is it is actually having a problem connecting to the elasticsearch container. I proceeded to manually test the container and everything seemed fine. The problem was in the spring config file I used I called the host localhost instead of localhost:9200 with the port. I updated the application.properties or application.yaml file to include the right port and viola I was able to connect.

spring:
  datasource:
    url: jdbc:mysql://localhost:3306/recisphere?autoReconnect=true&useUnicode=true&characterEncoding=UTF-8&allowMultiQueries=true&useSSL=false
    username: test
    password: test
  jpa:
    generate-ddl: true
    properties:
      hibernate:
        dialect: org.hibernate.dialect.MySQL5InnoDBDialect
        ddl-auto: create
        search:
          backend:
            hosts: localhost:9200

 

Posted in Elasticsearch, Java | No Comments »
Tagged: , ,

Move from Quay to AWS ECR Automation

Posted on July 19, 2021 by

Why the move

We recently have decided to move all the docker repositories from a private registry called Quay to ECR. Quay uses robot tokens in order to authenticate the registry in order to push images. We have found it is more cost effective to just use ECR instead of pay for and maintain Quay Registry.

Automate Jenkins

All of our builds have a docker step in the pipe that builds the docker images after we run a combative set of tests. Here is the code in a bash script that we use to automate the push to ECR. This will create the container registry automatically if it doesn’t exist and then pushes the changes to it. Because elastic container registry uses IAM roles we also use amazon-ecr-credential-helper in order to manage the fact we have multiple accounts with AWS. We use kustomize and grab the namespace from there in order to name the docker registry. Then we also need to update the policy for ECR to allow other accounts need access to the image and the easiest way is to just allow the other accounts access.

namespace="$(kubectl kustomize k8/base/ | grep namespace | head -1 | awk 'NF>1{print $NF}')"
#aws --region us-east-1 ecr describe-repositories --repository-names ${namespace} || aws --region us-east-1 ecr create-repository --repository-name ${namespace} --image-tag-mutability IMMUTABLE --tags Key=StackId,Value=${namespace} --image-scanning-configuration scanOnPush=true
aws --region us-east-1 ecr describe-repositories --repository-names ${namespace} || aws --region us-east-1 ecr create-repository --repository-name ${namespace} --image-tag-mutability MUTABLE --tags Key=StackId,Value=${namespace} --image-scanning-configuration scanOnPush=true
REGISTRY_PATH="${REGISTRY_BASE}/${namespace}"


####
# SET PERMISSIONS
####

cat <<'EOF' > policy.json
{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "ReadOnlyWithinAncestry",
      "Effect": "Allow",
      "Principal": "*",
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchGetImage",
        "ecr:GetAuthorizationToken",
        "ecr:GetDownloadUrlForLayer"
      ]
    },
    {
      "Sid": "AllowCrossAccount",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "ecr:*"
    }
  ]
}
EOF
aws --region us-east-1 ecr set-repository-policy --repository-name ${namespace} --policy-text file://policy.json


echo "Building Docker Image"
echo ${REGISTRY_PATH}:${buildNumber}

#This builds the container and tags it locally
#aws --region us-east-1 ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 310865762107.dkr.ecr.us-east-1.amazonaws.com
docker build --pull -t ${REGISTRY_PATH}:${buildNumber} .

echo "Publishing now"

echo "Publishing since running on the build machine."

#If PUBLISH has been set we assume we are on the build machine and will also push the image to the registry.
docker push ${REGISTRY_PATH}:${buildNumber}
echo "Push latest"

# tag/push latest
#This is not needed as we now use aws credentials manager
#aws --region us-east-1 ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 878616621923.dkr.ecr.us-east-1.amazonaws.com
docker tag ${REGISTRY_PATH}:${buildNumber} ${REGISTRY_PATH}:latest
docker push ${REGISTRY_PATH}:latest

 

Posted in Amazon Web Services | No Comments »
Tagged: , , , ,

How to Set Master Nodes in Elasticsearch

Posted on June 3, 2020 by

Problem

In order to horizontally scale an elasticsearch cluster in the cloud we need to make sure we don’t remove the master nodes if we scale the cluster down during times of less usage. We do this in Amazon Web Services by having two groups of compute instances. The first block is equal to the number of shards in the cluster. We set a unique name on these so they can be easily found in AWS. We treat these as masters and never scale them down. Preserving the master nodes helps prevent data loss. Unfortunately elasticsearch doesn’t let us set master nodes and it doesn’t guarantee that which nodes in the cluster are the master nodes.

Solution

We need to force a set of shards onto the instances of our choice. The following is a script that queries aws based on a tag and then uses the reallocation endpoint in elasticsearch to swap shards.

fun forceUniqueness(ips: List<String>, numberShards: Int) {
        //check to see if we are unique
        //yes exit
        if (validateUniqueShards(ips)) {
            return
        } else {
            //no ->
            val fullMap = getShardMap()
            val fullList = getShardList()
            var duplicateMap = mutableMapOf<String, Int>()
            var duplicatedShards = mutableSetOf<Int>()
            var missingShards = mutableSetOf<Int>()
            var listMoveTo = mutableListOf<String>()

            var tmpSet = mutableSetOf<Int>()

            //which ones are duplicated
            for (ip in ips) {
                if (tmpSet.contains(fullMap.get(ip))) {
                    duplicateMap.put(ip, fullMap.getValue(ip))
                    duplicatedShards.add(fullMap.getValue(ip))
                    listMoveTo.add(ip)
                } else {
                    tmpSet.add(fullMap.get(ip)!!)
                }
            }

            //which ones are missing
            for (i in 0..numberShards - 1) {
                if (!tmpSet.contains(i)) {
                    missingShards.add(i)
//                    println(i)
                }
            }

            println("FoundShards")
            println(tmpSet)
            println("MissingShards")
            println(missingShards)
            println("DuplicateMap")
            println(duplicateMap)

            //find a shard with missing and move it over
            var tripleList = mutableListOf<Triple<String, String, Int>>()
            for (shard in missingShards) {
                for (json in fullList) {
                    //if it is in an ignore ip list because of multiple on demand data integrity lines continue
                    if (ignoreIps.contains(json.getString("ip"))) {
                        continue
                    }
                    if (json.getInt("shard") == shard &&  json.getString("prirep") == "r") {

                        var moveFrom = json.getString("node")
                        var moveTo = getNodeName(listMoveTo.removeAt(0))
                        tripleList.add(Triple(moveFrom, moveTo, shard))

                        //todo move the one that is duplicated and swap
                        var tmpShard = getShardByNode(moveTo)
                        tripleList.add((Triple(moveTo, moveFrom, tmpShard)))
                        break
                    }
                }
            }
            moveShards(tripleList)
        }
    }

 

Posted in Uncategorized | No Comments »
Tagged: , , , , , ,