Lofland bLOG

The permissions on a folder used as a mount point, and the permissions on the top of the filesystem itself are not related.

In other words. If you ls on a directory and then mount a filesystem to that directory and run the same ls command, the permissions will look different.

Thus:

1. Restrict the permissions on the mount point (directory) to be read only. This way, if the filesystem is not mounted for some reason no one can write any files to the directory. This will prevent two things. First, it will prevent someone from filling up the root filesystem by accident. Second, it will prevent files from getting spread out across the directory (inside the mount point) and the filesystem itself. It can be a real mess trying to resync up a filesystem and the files stuck in the directory.

2. When creating a new filesystem, be sure to mount it, and set the permissions, since you can’t do this without the filesystem being mounted.

How do you tell if a directory is just that, or if it is it’s own filesystem? df -k :

root> df -k /home
/home (/dev/vg00/lvol5 ) : 20464 total allocated Kb
8728 free allocated Kb
11736 used allocated Kb
57 % allocation used
root> df -k /etc
/ (/dev/vg00/lvol3 ) : 143360 total allocated Kb
3976 free allocated Kb
139384 used allocated Kb
97 % allocation used
See, /home is its own filesystem, but /etc is in /, as you can see above.

What do you do if you have a directory full of files that should have been in a filesystem mounted at that point?

First, mount the filesystem manually to another directory. Under temp or something.

Use fuser to see if any of the files are open. They must not be.

Then do something like this to get the files over to the mount point.

cd /directory_that_should_not_have_files_in_it

find . -depth -xdev | cpio -dumpx /tmp/mount_point

man find so you can verify my switches here (HPUX). These find switches say to traverse the directory tree first (-depth) and don’t cross a mount point (-xdev). The cpio switches say -d make the directories first, -u unconditionally -m retain modify times -p pass-through from std in and -x save or restore special files.

Now unmount the filesystem from the temp location and mount it properly. Oh! First fix the permissions on that directory so that next time the application or user cannot write to it!

lsof will tell you who is using what file system. It is a pretty standard unix util, but not all OS’s install it by default. There is a package to install it on HPUX

man lsof

fuser can also be used to find out what proccess has a file open on a file system

man fuser

cd /var
bdf . = free space just here

cleanup -c 1 = clean up any patches that have been superceded at least ome time. May free up some space

check /var/tombstones
check /var/adm/wtmp = last user login info, type “last” to get the info, most systems zero this out daily

Use this to list how much space is used by each directory, starting with the biggest. Useful for spotting where the problem may be:
du -kxa . | sort -rn | more

remember:

> filename = zero a file out, used for open files! (same as cat /dev/null > filename)

/var/adm/wtmp holds the info that “last” shows It gets quite huge if noone cleans it out. Most boxes have a cron job to do this just:
> /var/adm/wtmp to empty it

last = last good login
lastb = last login failure

Remember to cp things out to another filesystem and gzip them before zeroing them out if you are unsure about whether they will be needed later or not.

I’ve patched this together, some of my facts may be wrong, but I think the solution is right.
When an application is writing to a file, it may keep the file open. The application will “grab” the file and open a “handle.” If you delete the file, the application still has an open handle to the file’s “inode.” Deleting or moving the file out from under the application can have some bad affects. One possibility is that the applicatoin just keeps writing to the same handle, and the space is not cleared from the drive. So if your drive was filling up, it just keeps filling. Another possibility is that the application just gets upset and will not log anymore until you restart it.

The solution is to “zero” the file instead by simply running:

cat /dev/null > file_name

this has become such a common thing to do that on most Unix flavors this does the same thing:

> file_name

That will “zero” the file so that it is empty, but leave the inode alone, and the open handle that the applicatoin has open will still work, but now your disk wil have more free space again!

If you need to preserve the file’s contents, use the cp command to copy the file to somewhere else first befire you “zero” it. Then gzip the copy. Just don’t ever use mv or rm on an open file!

The best thing to do is stop the application, then move the log, then restart it, but usually you don’t want to do that, and besides, isn’t “uptime” what unix is all about? Also, some application simply open the file for each log entry and then close it again, in which case you can rm or mv the file, but knowing that is more difficult than just using the “zero” method above

See page 495 of “HPUX CSA” by Rafeeq Rehman
This entry contains notes from training by coworkers, personal experience, and the above mentioned book. I do not claim any of it to be original with me, unless you see a mistake. I am sure the mistakes are mine.

Take a look at your disks. An easy way is to use bdf. This is kind of the HPUX equivalent of df.

vgdisplay will show all of the existing volume groups, and vgdisplay -v will give the details about the logical and physical volumes in the group.

Each volume group has a unique directory under /dev/ where the LVM device files are kept.
Here we call them vg*

/dev/vg00 will usually be the volume for the onboard disks, where HPUX lives. The other vg’s will probably exist on drives from an SAN

Each /dev/vg*/ directory contains three things:

A group file.
- It is a “c” (character special) file
- Each volume group needs one of these files. It is created before the vg is actually built
- It must have a unique “minor number.” The minor number is a hexidcimal number starting with 0x in the long listing. The first two digits are the vg #. The last four digits are always 0’s. If you name your vg groups vg01, vg02, vg03, etc, then you can make the minor number correspond to the vg name. Otherwise I’m not sure that there is any direct correlation.
- The major number for all LVM device files is 64
root:/dev/vg00> ll group
crwxr-xr-x 1 root sys 64 0×000000 Dec 12 2003 group

The minor number for the group file must be unique for each vg (it is a hex number). You only work with the first two digits, the last four are always 0000.
A “b” (block device) file for each logical volume
- This is the device you mount

A “c” (character special) file for each logical volume called rlvolname
-This is the raw device, used to format the filesystem

Each logical volume device file will have the major number 64, and the minor number wil be the same as the group file’s, but with the last digits showing the lv number. Again, like the vg#’s, they may or may not correspond to the names (LV01, LV02, etc.), but they will be sequential. See pg 499 of the HPUX CSA book.
You can see the various mount pounts (logical volumes) for HPUX under /dev/vg00
- Remember, group is not a lvol, it is a special file for the vg

The first thing you have to do before making a new vg is find out what the 0x numbers are for the goup file in each volume group, so that we can make a new one.

To find all of them:
find /dev -name group -exec ls -al {} \;

root:/root> find /dev -name group -exec ls -al {} \;
crwxr-xr-x 1 root sys 64 0×000000 Dec 12 2003 /dev/vg00/group
crw——- 1 root sys 64 0×040000 Mar 8 11:04 /dev/vgfapp/group
crw——- 1 root sys 64 0×050000 Mar 8 11:04 /dev/vgfdata/group

or just typing this usually works too:
ll /dev/*/group
(ll = ls -la on non HPUX machines)

So now we can use 0×020000, 0×030000, or 0×060000 and above. They don’t have to be sequential, but there isn’t any point in using random numbers here.

mkdir /dev/vg02

man vgcreate: look under examples, the mkdir and mknod command is right there

mknod /dev/vg02/group c 64 0×020000

We are telling it to make a character special file, with a major number of 64 (all volume groups have a major number of 64) and the minor number of 0×020000 (which we explained above).

Now we need to find some free hard drives (disks) to use:

The disks on the system are listed here:
ls /dev/dsk/*

To list all of the existing volume groups, and show what disks they are using you can run this:
vgdisplay -v

ioscan -funCdisk shows a bunch of disks, but how do we know how/when/where they are used? Plus with a SAN, there is a primary and an alternate path to each disk (they look like two disks to the system, but they are not), so the SAN disks show twice. You really only have half of the number listed.

Again, how do I know if they are bing used? - HP has no good way of telling you waht is not used.

ls /dev/dsk/* will tell us what disks are available to HPUX
for i in `ls /dev/dsk/*`; do pvdisplay $i; done 1>/dev/null
-so the errors are the drives that are not in a volume group (we piped the stin to null and just watched stderr)

or

To get a good list of available drives:
for i in `ls /dev/dsk/*`; do pvdisplay $i; done 2>/tmp/disk.out
cat /tmp/disk.out | grep -v Could | grep -v belongs | cut -d’ ‘ -f6 | sed s/\”//g | sed s/\.$//g > /tmp/disklist
(rm /tmp/disk.out)

Ok, now /tmp/disklist has a list of available disks.
Just to double check our list:

for i in `cat /tmp/disk2/`;do pvdisplay $i;done 2> /dev/null

-We piped error to null, but ALL should return errors, thus Should be BLANK, meaning everything in the list was NOT a volume

(Remember, all disks on an HP box are in a vg if they are used.)

Now we know what disks are not being used, however, half of those are alternate links. How do we know which ones?

EMC SAN Arrays:

inq -sortsymm = this is an EMC tool, so it works if we have EMC disks in the box Each disk has a serial number. The first 3 digits (220) is the frame serial number, next two E8 is the disk, last part is the frame. So here you can see what drives are the same drive. The Array usually has a bunch of 72 gig drives, but they are shown to us as 8 gig drives. Internal drives will always be a d0, b/c they are not carved up.

Usually with EMC disks, the C#’s are different but the t and d #’s are the same for the same drive on the EMC. It doesn’t have to be that way though, so watch those serial numbers.

Each EMC drive has a unique serial #, so you can see if two disks on the local system are just redundant paths to the same EMC path.

To combine the free disk list from HPUX with the serial number list from the Hitachi SAN, and get a list of availalbe drives with serial numbers:

for i in `ls /dev/dsk/*`; do pvdisplay $i; done 2>/tmp/disk.out
cat /tmp/disk.out | grep -v Could | grep -v belongs | cut -d’ ‘ -f6 | sed s/\”//g | sed s/\.$//g > /tmp/disklist
rm /tmp/disk.out

inq -sortsymm | grep “rdsk” > /tmp/seriallist
for i in kk`cat /tmp/disklist | cut -d”/” -f4`; do grep $i /tmp/seriallist;done | sort -k 5 > /tmp/availabledisklist
rm /tmp/disklist
rm /tmp/seriallist
Echo Your list is in “/tmp/availabledisklist”
#Maybe we could combine a few more commands and ellimnate one more temp file, but the commands start to get insane

Obviously the DVD-ROM isn’t “available” and any other “local” drives are not what we want. The list should clearly show if they are EMC drives or something else.

Also, there are what may be EMC “control” disks, they are small 4 meg, and 2 meg drives. Don’t use these. Just use the ones that are in the standard 8gig size.

Hitachi SAN Arrays:

lunstat -ts | egrep “Serial|Device” | paste -s -d”\t\n” -

This will list the devices with the serial numbers, so you can see which drives are identical.
(Be sure to sanity check it all. The ctd numbers should be similar for same drives, when you add drives to the volume, it should automatically recognize them as “alernate paths” to the same drive.

Don’t use drives that are not on the SAN. They will have different serial numbers.

To combine the free disk list from HPUX with the serial number list from the Hitachi SAN, and get a list of availalbe drives with serial numbers:

for i in `ls /dev/dsk/*`; do pvdisplay $i; done 2>/tmp/disk.out
cat /tmp/disk.out | grep -v Could | grep -v belongs | cut -d’ ‘ -f6 | sed s/\”//g | sed s/\.$//g > /tmp/disklist
rm /tmp/disk.out

lunstat -ts | egrep “Serial|Device|Manufacturer” | paste -s -d”\t\t\n” - > /tmp/seriallist
for i in `cat /tmp/disklist | cut -d”/” -f4`; do grep $i /tmp/seriallist;done | sort -k 9 > /tmp/availabledisklist
rm /tmp/disklist
rm /tmp/seriallist
Echo Your list is in “/tmp/availabledisklist”
#Maybe we could combine a few more commands and ellimnate one more temp file, but the commands start to get insane

The list should clearly show if they are Hitachi drives or something else. We don’t want the drives made by “HP,” etc. They are either internal disks, or things like the DVD-ROM. You can always check on them with ioscan -funCdisk

(Note, that the Hitachi lunstat program works on EMC arrays, so you can use it if you want. However, the EMC inq program does not report the serial number on Hitachi drives, so you it will not work for them.)

Now, to create a volume group:

pvcreate /dev/rdsk/c29t12d2 -> told us it already belongs a to a volume group
pvdisplay /dev/dsk/c29t12d2 -> says no volume group
-problem is, the drive WAS a member of a volume group, but isn’t anymore
–if you vgreduce this drive out of the volume group before you export the vg, this probably won’t happen
-if you are SURE it is not used anymore then:
pvcreate -f /dev/rdsk/c29t12d2 - DANGER! this will wipe out a drive, very careful! the -f forces it.

man vgcreate
vgcreate -e 30000 vg02 /dev/dsk/c29t12d2 - must give it at least one drive.

NOTE: -e sets the Max PE per PV Once you create the volume group and use the disk, by default whatever size that disk is, that is the largets disk you can add by default. If you built it with a 4 gig disk and then later added a 20gig disk, you could only use 4 gigs on the 20gig disk. Instead, we say, give me X “PE Size (Mbytes) extents = Max PE per PV. John always uses 30000 for Max PE per PV, with PE Size (Mbytes) of 4 (default PE size).

vgdisplay (and there it is!) (size is Total PE x PE Size)

Now find the other drive with the same serial number and add it, it should show up as an “alternate link”

vgextend vg02 /dev/dsk/c27t12d2
- It figures out that this is the same disk and adds it as an “Alternate Link”
- You only need to use pvcreate on the disk once, not once for each “link”
vgkk
pvcreate /dev/rdsk/c29t12d3
pvdispaly /dev/rdsk/c29t12d3
pvcreate -f /dev/rdsk/c29t12d3
vgextend vg02 /dev/dsk/c29t12d3
vgreduce vg02 /dev/dsk/c29t12d3 (oops!)
vgextend vg02 /dev/dsk/c27t12d3 (Swapp controllers, so that second drive in the vg has the other controller as primary)
vgextend vg02 /dev/dks/c29t12d3
vgdispay -v
- now primary controller is alternated back and forth, b/c HPUX goes to primary first, then to backup, so this allows the system to use both controllers.

NOTE: We could just put all of the PV names on the same command line with the first vgcreate command, rather than using a bunch of vgextend commands. vgcreate will take the names of multiple physical volumes. A vgextend command will also probably take multiple pv names.

man lvcreate
(size is L or l by size or le number)
lvcreate -L 1000m -n lvol1 vg02
vgdisplay -v (now some disk space is missing from PV’s and there is an LV)

(try to mount it)
cd /
mkdir chris
mount /dev/vg02/lvol1 /chris
(NOT FORMATTED! )

newfs -F vxfs /dev/vg02/rlvol2
(newfs is a front end for mkfs)
mount /dev/vg02/lvol1 /chris
bdf
It worked!

Set permissions on the filesystem after you mount it. Setting them on the mount directory first doesn’t do any good.

(now make it 2000m)

lvextend -L 2000m /dev/vg02/lvol1

vgdisplay -v
(now it is bigger)

bdf
(not bigger )
We gave the lvol more, but not the filessytem

fsadm -b 2000m /chris

Now the FS has been expanded in place, online

(man pages for fsadm are incorrect, what we did is not in there)

swlist | grep -i online
- fsadm to do this is called “onlinejfs”

DIG FOR SOME FSADM documentation!

mount - will show it is there
vi /etc/fstab
add
/dev/vg02/lvol1 /chris vxfs delaylog 0 2 (0 & 2 is for when to check for dirty bit, 1 = at boot 2= later)

umount /chris

now you can just type
mount /chris
and it knows where to mount it

to remove a volume use vgexport

umount /chris
vgdisplay -v
(good to back it down)
(Remove alternate links first, if you do pri first, it just switches alt to pri, doesnt’ really matter, but…)
lvremove /dev/vg02/lvol1
vgreduce vg02 /dev/dsk/c27t12d3
vgreduce vg02 /dev/dsk/c29t12d3
(I don’t think you can reduce the last disk out of the volume group)
vgchange -a n vg02
vgexport vg02 (dn’t be in /dev/vg02 when you do this or it won’t remove the directory)
(gone)
vgdisplay -v

Create a mirrored volume?

lvextend -m 1 /dev/vgora/lvolora

-But that won’t work if we only have one pv in the vg! You can turn off strict, but it is really silly.

pvcreate /dev/rdsk/c35t9d2
vgextend vgora /dev/dsk/c35t9d2
vgextend vgora /dev/dsk/c34t9d2

lvextend -m 1 /dev/vgora/lvolora

Make a stripe set?

lvcreate -L 1000m -n lvolstripeset -i 3 -I 128 vgora